Sep 18 16:05:02.239538 ip-10-0-135-253 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 16:05:02.641203 ip-10-0-135-253 kubenswrapper[2546]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:05:02.641203 ip-10-0-135-253 kubenswrapper[2546]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 16:05:02.641203 ip-10-0-135-253 kubenswrapper[2546]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:05:02.641203 ip-10-0-135-253 kubenswrapper[2546]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 16:05:02.641203 ip-10-0-135-253 kubenswrapper[2546]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:05:02.642367 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.642283    2546 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 16:05:02.648068 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648051    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:05:02.648068 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648067    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648070    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648073    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648077    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648079    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648082    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648085    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648088    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648090    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648093    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648098    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648102    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648106    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648116    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648119    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648122    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648125    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648127    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648130    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648133    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:05:02.648139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648136    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648139    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648142    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648145    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648148    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648152    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648155    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648158    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648161    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648164    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648166    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648169    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648173    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648177    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648180    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648183    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648185    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648188    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648190    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:05:02.648629 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648196    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648199    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648201    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648204    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648206    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648209    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648212    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648215    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648217    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648221    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648225    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648228    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648231    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648233    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648236    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648251    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648254    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648256    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648259    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648262    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:05:02.649088 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648264    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648267    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648270    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648272    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648275    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648277    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648280    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648283    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648288    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648291    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648293    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648296    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648299    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648305    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648308    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648310    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648313    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648316    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648318    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648321    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:05:02.649585 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648323    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648326    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648329    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648332    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648334    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648337    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648339    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648342    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648345    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648348    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648350    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648353    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648356    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648359    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648361    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648364    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648366    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648369    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.648372    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649065    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:05:02.650062 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649071    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649074    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649076    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649079    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649082    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649084    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649091    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649094    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649096    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649100    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649102    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649107    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649110    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649114    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649118    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649121    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649124    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649126    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649129    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:05:02.650625 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649132    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649136    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649139    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649142    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649144    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649147    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649149    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649152    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649155    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649158    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649160    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649163    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649165    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649168    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649170    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649172    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649175    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649178    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649181    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:05:02.651116 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649183    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649191    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649194    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649196    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649199    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649202    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649204    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649207    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649209    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649212    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649214    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649217    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649220    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649224    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649227    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649230    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649232    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649235    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649251    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649254    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:05:02.651603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649257    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649259    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649262    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649265    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649267    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649270    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649272    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649275    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649278    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649280    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649283    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649285    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649288    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649290    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649299    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649302    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649304    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649307    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649309    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649312    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:05:02.652170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649315    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649317    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649319    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649322    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649324    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649329    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649332    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649335    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649338    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649341    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649343    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649346    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649348    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649352    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649355    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649358    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649361    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649363    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649366    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:05:02.652683 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649369    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649750    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649755    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649758    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649760    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649763    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649766    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649769    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649772    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649774    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649777    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649780    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649782    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649785    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649788    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649790    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649793    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649795    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649798    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649801    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:05:02.653149 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649804    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649806    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649809    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649812    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649814    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649817    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649820    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649825    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649828    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649830    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649833    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649836    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649838    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649841    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649843    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649846    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649848    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649851    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649854    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:05:02.653649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649856    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649860    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649862    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649866    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649868    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649871    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649873    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649876    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649878    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649881    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649884    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649887    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649889    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649892    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649895    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649897    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649900    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649903    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649905    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649908    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:05:02.654099 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649912    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649915    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649919    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649923    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649926    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649929    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649931    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649934    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649937    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649940    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649942    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649945    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649948    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649950    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649954    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649959    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649962    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649965    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649968    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:05:02.654651 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649971    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649973    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649976    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649980    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649983    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649985    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649988    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649990    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649993    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649996    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.649999    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650002    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650005    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650009    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650011    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650014    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650016    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650019    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650022    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:05:02.655146 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650024    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650027    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650029    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650105    2546 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650147    2546 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650154    2546 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650159    2546 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650163    2546 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650167    2546 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650171    2546 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650176    2546 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650179    2546 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650182    2546 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650185    2546 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650188    2546 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650191    2546 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650195    2546 flags.go:64] FLAG: --cgroup-root=""
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650198    2546 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650202    2546 flags.go:64] FLAG: --client-ca-file=""
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650205    2546 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650208    2546 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650212    2546 flags.go:64] FLAG: --cluster-domain=""
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650214    2546 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650218    2546 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 16:05:02.655618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650221    2546 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650224    2546 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650229    2546 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650232    2546 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650235    2546 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650252    2546 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650255    2546 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650258    2546 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650261    2546 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650264    2546 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650267    2546 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650271    2546 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650274    2546 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650277    2546 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650280    2546 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650283    2546 flags.go:64] FLAG: --enable-server="true"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650286    2546 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650291    2546 flags.go:64] FLAG: --event-burst="100"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650295    2546 flags.go:64] FLAG: --event-qps="50"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650297    2546 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650301    2546 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650303    2546 flags.go:64] FLAG: --eviction-hard=""
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650307    2546 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650310    2546 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650313    2546 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 16:05:02.656200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650316    2546 flags.go:64] FLAG: --eviction-soft=""
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650319    2546 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650322    2546 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650325    2546 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650328    2546 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650331    2546 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650334    2546 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650337    2546 flags.go:64] FLAG: --feature-gates=""
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650341    2546 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650345    2546 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650348    2546 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650351    2546 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650355    2546 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650358    2546 flags.go:64] FLAG: --help="false"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650361    2546 flags.go:64] FLAG: --hostname-override="ip-10-0-135-253.ec2.internal"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650364    2546 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650367    2546 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650370    2546 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650373    2546 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650377    2546 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650379    2546 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650382    2546 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650385    2546 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 16:05:02.656806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650388    2546 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650391    2546 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650394    2546 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650397    2546 flags.go:64] FLAG: --kube-reserved=""
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650400    2546 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650404    2546 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650407    2546 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650409    2546 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650413    2546 flags.go:64] FLAG: --lock-file=""
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650416    2546 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650419    2546 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650422    2546 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650427    2546 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650430    2546 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650433    2546 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650436    2546 flags.go:64] FLAG: --logging-format="text"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650438    2546 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650442    2546 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650446    2546 flags.go:64] FLAG: --manifest-url=""
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650449    2546 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650453    2546 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650456    2546 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650460    2546 flags.go:64] FLAG: --max-pods="110"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650464    2546 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650467    2546 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 16:05:02.657467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650470    2546 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650473    2546 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650475    2546 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650478    2546 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650481    2546 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650493    2546 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650496    2546 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650499    2546 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650502    2546 flags.go:64] FLAG: --pod-cidr=""
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650505    2546 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650511    2546 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650515    2546 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650518    2546 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650521    2546 flags.go:64] FLAG: --port="10250"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650524    2546 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650527    2546 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09ff4718803d7dd41"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650530    2546 flags.go:64] FLAG: --qos-reserved=""
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650533    2546 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650536    2546 flags.go:64] FLAG: --register-node="true"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650539    2546 flags.go:64] FLAG: --register-with-taints=""
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650546    2546 flags.go:64] FLAG: --registry-burst="10"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650549    2546 flags.go:64] FLAG: --registry-qps="5"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650552    2546 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650555    2546 flags.go:64] FLAG: --reserved-memory=""
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650558    2546 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 16:05:02.658085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650563    2546 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650566    2546 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650569    2546 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650572    2546 flags.go:64] FLAG: --runonce="false"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650575    2546 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650578    2546 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650581    2546 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650584    2546 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650587    2546 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650590    2546 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650593    2546 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650596    2546 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650599    2546 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650601    2546 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650604    2546 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650607    2546 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650610    2546 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650613    2546 flags.go:64] FLAG: --system-cgroups=""
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650616    2546 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650622    2546 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650625    2546 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650628    2546 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650632    2546 flags.go:64] FLAG: --tls-min-version=""
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650638    2546 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650641    2546 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 16:05:02.658743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650645    2546 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650648    2546 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650651    2546 flags.go:64] FLAG: --v="2"
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650655    2546 flags.go:64] FLAG: --version="false"
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650659    2546 flags.go:64] FLAG: --vmodule=""
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650662    2546 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.650666    2546 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650783    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650787    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650791    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650794    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650797    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650800    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650803    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650807    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650810    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650813    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650816    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650819    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650821    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650824    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:05:02.659366 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650827    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650829    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650832    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650835    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650838    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650841    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650846    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650849    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650852    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650856    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650859    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650861    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650864    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650866    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650869    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650872    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650874    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650877    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650880    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650883    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:05:02.659869 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650886    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650888    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650891    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650894    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650896    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650899    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650902    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650904    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650907    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650909    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650912    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650915    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650917    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650920    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650923    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650926    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650928    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650931    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650933    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650937    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:05:02.660360 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650940    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650944    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650946    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650949    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650952    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650954    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650957    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650959    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650962    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650965    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650968    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650970    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650974    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650978    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650981    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650984    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650986    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650989    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650991    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650994    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:05:02.660870 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.650997    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651000    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651002    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651005    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651007    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651010    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651013    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651015    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651018    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651022    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651025    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651028    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651031    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651035    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651038    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651041    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651044    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651046    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651049    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:05:02.661766 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651052    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651054    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651057    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651060    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651063    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.651066    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.651676    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.659261    2546 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.659277    2546 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.660698    2546 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.660707    2546 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661595    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661631    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661637    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661642    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:05:02.662351 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661648    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661688    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661696    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661701    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661706    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661710    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661714    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661718    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661723    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661727    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661731    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661736    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661740    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661749    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661754    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661758    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661762    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661767    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661771    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:05:02.662763 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661775    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661780    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661784    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661789    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661793    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661797    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661801    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661810    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661814    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661818    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661850    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661854    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661861    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661868    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661872    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661877    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661881    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661886    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661890    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:05:02.663213 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661900    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661907    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661913    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661917    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661923    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661927    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661932    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.661940    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662222    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662262    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662267    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662271    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662274    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662277    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662280    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662284    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662287    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662290    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662292    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662295    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:05:02.663700 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662298    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662301    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662303    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662306    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662309    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662311    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662315    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662318    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662321    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662323    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662326    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662330    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662333    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662336    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662339    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662342    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662345    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662348    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662351    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662355    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:05:02.664194 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662358    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662361    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662364    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662367    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662370    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662372    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662375    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662377    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662380    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662383    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662385    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662388    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662391    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662394    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662398    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662403    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:05:02.664738 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662406    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.662411    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662544    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662549    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662552    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662554    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662557    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662560    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662563    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662565    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662568    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662571    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662574    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662577    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:05:02.665143 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662579    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662582    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662586    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662590    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662593    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662596    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662599    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662602    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662605    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662608    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662611    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662614    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662617    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662619    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662622    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662625    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662628    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662631    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662633    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662636    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:05:02.665554 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662639    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662641    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662644    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662647    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662649    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662652    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662656    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662659    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662662    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662665    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662668    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662671    2546 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662674    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662677    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662679    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662682    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662685    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662689    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662692    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662695    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:05:02.666045 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662698    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662701    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662703    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662706    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662709    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662711    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662714    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662717    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662719    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662722    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662725    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662727    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662730    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662733    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662735    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662738    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662741    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662743    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662746    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662748    2546 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:05:02.666557 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662751    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662754    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662756    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662760    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662764    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662767    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662770    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662773    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662776    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662779    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662782    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662786    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662789    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662791    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662794    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662797    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662799    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662802    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662805    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:05:02.667029 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662807    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:05:02.667502 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662810    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:05:02.667502 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662812    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:05:02.667502 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662815    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:05:02.667502 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662818    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:05:02.667502 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662820    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:05:02.667502 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662823    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:05:02.667502 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:02.662825    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:05:02.667502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.662830    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:05:02.667502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.662933    2546 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 16:05:02.667502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.664946    2546 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 16:05:02.667502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.665773    2546 server.go:1019] "Starting client certificate rotation"
Sep 18 16:05:02.667502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.665870    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:05:02.667502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.665906    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:05:02.669090 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.669076    2546 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:05:02.670217 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.670200    2546 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:05:02.680480 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.680461    2546 log.go:25] "Validated CRI v1 runtime API"
Sep 18 16:05:02.684213 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.684195    2546 log.go:25] "Validated CRI v1 image API"
Sep 18 16:05:02.686567 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.686553    2546 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 16:05:02.690575 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.690555    2546 fs.go:135] Filesystem UUIDs: map[73097a3a-a1ea-490c-9a73-95c0baf65da9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e9e4a9ee-d7d5-4e0a-9e40-5bc6bb86f86a:/dev/nvme0n1p4]
Sep 18 16:05:02.690646 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.690575    2546 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 16:05:02.696222 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.696207    2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:05:02.696705 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.696582    2546 manager.go:216] Machine: {Timestamp:2026-09-18 16:05:02.695376664 +0000 UTC m=+0.339940636 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109370 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec259d897a180dc0ce3c144943d19adf SystemUUID:ec259d89-7a18-0dc0-ce3c-144943d19adf BootID:33a574af-68c1-4cba-84bd-229620aee0db Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:0e:c8:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:0e:c8:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:d3:9a:6e:8a:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 16:05:02.696802 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.696707    2546 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 16:05:02.696837 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.696812    2546 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 16:05:02.698382 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.698354    2546 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 16:05:02.698520 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.698385    2546 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-253.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 16:05:02.699038 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.699029    2546 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 16:05:02.699066 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.699041    2546 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 16:05:02.699066 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.699048    2546 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:05:02.699066 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.699063    2546 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:05:02.699149 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.699129    2546 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 16:05:02.703508 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.703496    2546 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:05:02.703629 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.703619    2546 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 16:05:02.703663 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.703656    2546 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 16:05:02.703700 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.703670    2546 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 16:05:02.703700 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.703683    2546 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 16:05:02.703700 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.703692    2546 kubelet.go:398] "Adding apiserver pod source"
Sep 18 16:05:02.703773 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.703710    2546 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 16:05:02.704274 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.704229    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:05:02.704351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.704282    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:05:02.709351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.709282    2546 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 16:05:02.711359 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.711340    2546 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 16:05:02.711428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.711372    2546 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 16:05:02.712563 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.712546    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 16:05:02.712563 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.712564    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 16:05:02.712680 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.712570    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 16:05:02.712680 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.712577    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 16:05:02.712680 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.712589    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 16:05:02.712680 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.712597    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 16:05:02.712680 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.712614    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 16:05:02.712680 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.712620    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 16:05:02.712680 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.712626    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 16:05:02.712680 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.712631    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 16:05:02.712680 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.712640    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 16:05:02.712680 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.712645    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 16:05:02.713309 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.713300    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 16:05:02.713309 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.713309    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 16:05:02.714956 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.714923    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:05:02.715035 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.714920    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:05:02.717300 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.717288    2546 server.go:1268] "Started kubelet"
Sep 18 16:05:02.717390 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.717368    2546 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 16:05:02.717453 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.717383    2546 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 16:05:02.717453 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.717445    2546 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 16:05:02.718151 ip-10-0-135-253 systemd[1]: Started Kubernetes Kubelet.
Sep 18 16:05:02.718950 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.718931    2546 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 16:05:02.719297 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.719280    2546 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 16:05:02.721552 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.721536    2546 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 16:05:02.722226 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.722094    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wg242"
Sep 18 16:05:02.724394 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.724369    2546 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 16:05:02.724489 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.724379    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 16:05:02.724489 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.724469    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:02.725340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.725322    2546 factory.go:55] Registering systemd factory
Sep 18 16:05:02.725340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.725341    2546 factory.go:223] Registration of the systemd container factory successfully
Sep 18 16:05:02.725485 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.725342    2546 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 16:05:02.725485 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.725357    2546 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 16:05:02.725485 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.725342    2546 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 16:05:02.725485 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.725440    2546 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 16:05:02.725485 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.725449    2546 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 16:05:02.725662 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.725578    2546 factory.go:153] Registering CRI-O factory
Sep 18 16:05:02.725662 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.725594    2546 factory.go:223] Registration of the crio container factory successfully
Sep 18 16:05:02.725662 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.725654    2546 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 16:05:02.725742 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.725668    2546 factory.go:103] Registering Raw factory
Sep 18 16:05:02.725742 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.725679    2546 manager.go:1196] Started watching for new ooms in manager
Sep 18 16:05:02.726255 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.726229    2546 manager.go:349] Starting recovery of all containers
Sep 18 16:05:02.731225 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.731207    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wg242"
Sep 18 16:05:02.734749 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.734677    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:05:02.734827 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.734801    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-253.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 16:05:02.735429 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.735414    2546 manager.go:354] Recovery completed
Sep 18 16:05:02.738416 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.734907    2546 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-253.ec2.internal.18d675df091adef9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-253.ec2.internal,UID:ip-10-0-135-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:05:02.717263609 +0000 UTC m=+0.361827558,LastTimestamp:2026-09-18 16:05:02.717263609 +0000 UTC m=+0.361827558,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:05:02.739858 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.739837    2546 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-253.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:05:02.745677 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.745664    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:02.747377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.747357    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:02.747447 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.747394    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:02.747447 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.747408    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:02.747942 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.747929    2546 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 16:05:02.747942 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.747942    2546 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 16:05:02.748036 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.747956    2546 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:05:02.748036 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.747935    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:02.748727 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.748713    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:02.748780 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.748739    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:02.748780 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.748751    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:02.749553 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.749490    2546 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-253.ec2.internal.18d675df0ae65d29  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-253.ec2.internal,UID:ip-10-0-135-253.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-253.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:05:02.747376937 +0000 UTC m=+0.391940899,LastTimestamp:2026-09-18 16:05:02.747376937 +0000 UTC m=+0.391940899,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:05:02.749950 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.749938    2546 policy_none.go:49] "None policy: Start"
Sep 18 16:05:02.749984 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.749954    2546 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 16:05:02.749984 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.749964    2546 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 16:05:02.751975 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.751963    2546 policy_none.go:47] "Start"
Sep 18 16:05:02.777814 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.777770    2546 manager.go:337] "Starting Device Plugin manager"
Sep 18 16:05:02.777814 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.777801    2546 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 16:05:02.777814 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.777809    2546 server.go:87] "Starting device plugin registration server"
Sep 18 16:05:02.801991 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.778020    2546 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 16:05:02.801991 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.778030    2546 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 16:05:02.801991 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.778123    2546 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 16:05:02.801991 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.778222    2546 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 16:05:02.801991 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.778268    2546 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 16:05:02.801991 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.778841    2546 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 16:05:02.801991 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.778890    2546 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:02.858974 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.858940    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 16:05:02.860113 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.860097    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 16:05:02.860183 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.860146    2546 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 16:05:02.860183 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.860169    2546 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 16:05:02.860276 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.860208    2546 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 16:05:02.862600 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.862585    2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:05:02.878291 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.878267    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:02.879074 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.879057    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:02.879135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.879085    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:02.879135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.879094    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:02.879135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.879116    2546 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-253.ec2.internal"
Sep 18 16:05:02.903122 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.903057    2546 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-253.ec2.internal"
Sep 18 16:05:02.903122 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.903079    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:02.915840 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.915823    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:02.961210 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.961191    2546 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-253.ec2.internal"]
Sep 18 16:05:02.961293 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.961263    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:02.961985 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.961967    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:02.962076 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.961996    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:02.962076 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.962010    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:02.964253 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.964225    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:02.964361 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.964347    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:02.964421 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.964386    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:02.964929 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.964912    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:02.965009 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.964931    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:02.965009 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.964944    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:02.965009 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.964948    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:02.965009 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.964959    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:02.965009 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.964961    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:02.966977 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.966965    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:02.967048 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.966988    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:05:02.967668 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.967653    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:05:02.967740 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.967683    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:05:02.967740 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:02.967697    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:05:02.990754 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.990733    2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-253.ec2.internal\" not found" node="ip-10-0-135-253.ec2.internal"
Sep 18 16:05:02.995051 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:02.995037    2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-253.ec2.internal\" not found" node="ip-10-0-135-253.ec2.internal"
Sep 18 16:05:03.016355 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:03.016336    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:03.027887 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.027867    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/257b3b74c0f4140f52501737bffd4b82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal\" (UID: \"257b3b74c0f4140f52501737bffd4b82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:03.027958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.027897    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/257b3b74c0f4140f52501737bffd4b82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal\" (UID: \"257b3b74c0f4140f52501737bffd4b82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:03.027958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.027922    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93cd48393ba29a5c2da55b21c3ff83e3-config\") pod \"kube-apiserver-proxy-ip-10-0-135-253.ec2.internal\" (UID: \"93cd48393ba29a5c2da55b21c3ff83e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:03.117128 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:03.117108    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:03.128625 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.128606    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/257b3b74c0f4140f52501737bffd4b82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal\" (UID: \"257b3b74c0f4140f52501737bffd4b82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:03.128709 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.128634    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/257b3b74c0f4140f52501737bffd4b82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal\" (UID: \"257b3b74c0f4140f52501737bffd4b82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:03.128709 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.128665    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93cd48393ba29a5c2da55b21c3ff83e3-config\") pod \"kube-apiserver-proxy-ip-10-0-135-253.ec2.internal\" (UID: \"93cd48393ba29a5c2da55b21c3ff83e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:03.128806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.128706    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/93cd48393ba29a5c2da55b21c3ff83e3-config\") pod \"kube-apiserver-proxy-ip-10-0-135-253.ec2.internal\" (UID: \"93cd48393ba29a5c2da55b21c3ff83e3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:03.128806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.128706    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/257b3b74c0f4140f52501737bffd4b82-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal\" (UID: \"257b3b74c0f4140f52501737bffd4b82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:03.128806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.128706    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/257b3b74c0f4140f52501737bffd4b82-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal\" (UID: \"257b3b74c0f4140f52501737bffd4b82\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:03.218017 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:03.217998    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:03.294549 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.294525    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:03.298478 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.298464    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:03.318417 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:03.318395    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:03.418934 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:03.418907    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:03.519488 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:03.519434    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:03.619959 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:03.619932    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:03.665458 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.665440    2546 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 16:05:03.665849 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.665559    2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 16:05:03.720891 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:03.720866    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:03.743119 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.743094    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 16:00:04 +0000 UTC" deadline="2028-06-07 18:49:28.919189084 +0000 UTC"
Sep 18 16:05:03.743119 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.743117    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15074h44m25.176075011s"
Sep 18 16:05:03.821585 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:03.821402    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:03.824208 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.824192    2546 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:05:03.896975 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:03.896952    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cd48393ba29a5c2da55b21c3ff83e3.slice/crio-390281c65d17d931f66e355cb3f4150606f36ea69566f334b58a3e7749efd1ca WatchSource:0}: Error finding container 390281c65d17d931f66e355cb3f4150606f36ea69566f334b58a3e7749efd1ca: Status 404 returned error can't find the container with id 390281c65d17d931f66e355cb3f4150606f36ea69566f334b58a3e7749efd1ca
Sep 18 16:05:03.902566 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.902554    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:05:03.916532 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:03.916517    2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:05:03.922586 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:03.922571    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:04.023150 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:04.023124    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-253.ec2.internal\" not found"
Sep 18 16:05:04.040408 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.040386    2546 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:05:04.125151 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.125109    2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:04.138163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.138147    2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:05:04.139133 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.139121    2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-253.ec2.internal"
Sep 18 16:05:04.151325 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.151310    2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:05:04.523696 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.523662    2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:05:04.544542 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:04.544510    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257b3b74c0f4140f52501737bffd4b82.slice/crio-53e8bc082cee604e4754bce3268a54e3608fc341153913f791d526d8ba2d3733 WatchSource:0}: Error finding container 53e8bc082cee604e4754bce3268a54e3608fc341153913f791d526d8ba2d3733: Status 404 returned error can't find the container with id 53e8bc082cee604e4754bce3268a54e3608fc341153913f791d526d8ba2d3733
Sep 18 16:05:04.705020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.704983    2546 apiserver.go:52] "Watching apiserver"
Sep 18 16:05:04.711981 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.711952    2546 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 16:05:04.714514 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.713674    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-p69sq","openshift-ovn-kubernetes/ovnkube-node-dnn6j","kube-system/kube-apiserver-proxy-ip-10-0-135-253.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2","openshift-multus/multus-additional-cni-plugins-qnmct","openshift-multus/multus-xxtjw","openshift-multus/network-metrics-daemon-7h8rh","openshift-network-operator/iptables-alerter-7xz8m","kube-system/global-pull-secret-syncer-pcmn7","kube-system/konnectivity-agent-js9rq","openshift-cluster-node-tuning-operator/tuned-jxv46","openshift-dns/node-resolver-g2bl6","openshift-image-registry/node-ca-6gkbl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal"]
Sep 18 16:05:04.716434 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.716411    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:04.716542 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:04.716481    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:04.719154 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.719138    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6gkbl"
Sep 18 16:05:04.721181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.721163    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7xz8m"
Sep 18 16:05:04.722169 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.722139    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 16:05:04.722169 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.722153    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tjzzn\""
Sep 18 16:05:04.722357 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.722143    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 16:05:04.722357 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.722299    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 16:05:04.723255 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.723230    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8cs6v\""
Sep 18 16:05:04.723549 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.723496    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.723639 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.723560    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:05:04.723826 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.723812    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 16:05:04.724456 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.724437    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 16:05:04.725157 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.725135    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 16:05:04.726105 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.726078    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 16:05:04.726105 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.726090    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.726232 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.726176    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 16:05:04.727223 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.726396    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 16:05:04.727223 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.726569    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hvq9z\""
Sep 18 16:05:04.728265 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.728228    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.728927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.728836    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 16:05:04.729276 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.729156    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 16:05:04.729276 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.729191    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 16:05:04.729599 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.729579    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vx59n\""
Sep 18 16:05:04.729718 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.729703    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 16:05:04.729810 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.729777    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 16:05:04.730492 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.730474    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdh8x\""
Sep 18 16:05:04.730571 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.730524    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 16:05:04.730630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.730574    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:04.730684 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:04.730651    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:04.732747 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.732728    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:04.732861 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:04.732799    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:04.735129 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.735110    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.737320 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.737306    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-js9rq"
Sep 18 16:05:04.738104 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738087    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 16:05:04.738213 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738159    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 16:05:04.738213 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738177    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 16:05:04.738213 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738212    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 16:05:04.738463 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738445    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h4v9d\""
Sep 18 16:05:04.738662 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738618    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvztn\" (UniqueName: \"kubernetes.io/projected/7ea7d464-a8e8-4c9e-b437-93784c147707-kube-api-access-qvztn\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.738705 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738680    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49ffc9f3-7c5f-48bc-8217-838ac6651979-host\") pod \"node-ca-6gkbl\" (UID: \"49ffc9f3-7c5f-48bc-8217-838ac6651979\") " pod="openshift-image-registry/node-ca-6gkbl"
Sep 18 16:05:04.738761 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738743    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/880072cc-3551-471f-b2f3-17b5a947dc82-iptables-alerter-script\") pod \"iptables-alerter-7xz8m\" (UID: \"880072cc-3551-471f-b2f3-17b5a947dc82\") " pod="openshift-network-operator/iptables-alerter-7xz8m"
Sep 18 16:05:04.738797 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738777    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlfxc\" (UniqueName: \"kubernetes.io/projected/880072cc-3551-471f-b2f3-17b5a947dc82-kube-api-access-qlfxc\") pod \"iptables-alerter-7xz8m\" (UID: \"880072cc-3551-471f-b2f3-17b5a947dc82\") " pod="openshift-network-operator/iptables-alerter-7xz8m"
Sep 18 16:05:04.738836 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738820    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wlpr\" (UniqueName: \"kubernetes.io/projected/3db26da6-d684-4099-88bd-68e1130faa42-kube-api-access-9wlpr\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.738872 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738846    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7ea7d464-a8e8-4c9e-b437-93784c147707-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.738919 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738871    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:04.738919 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738893    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49ffc9f3-7c5f-48bc-8217-838ac6651979-serviceca\") pod \"node-ca-6gkbl\" (UID: \"49ffc9f3-7c5f-48bc-8217-838ac6651979\") " pod="openshift-image-registry/node-ca-6gkbl"
Sep 18 16:05:04.739023 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738938    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/880072cc-3551-471f-b2f3-17b5a947dc82-host-slash\") pod \"iptables-alerter-7xz8m\" (UID: \"880072cc-3551-471f-b2f3-17b5a947dc82\") " pod="openshift-network-operator/iptables-alerter-7xz8m"
Sep 18 16:05:04.739023 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.738987    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-etc-selinux\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.739023 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739013    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-sys-fs\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.739174 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739048    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 16:05:04.739174 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739050    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ea7d464-a8e8-4c9e-b437-93784c147707-cnibin\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.739174 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739059    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 16:05:04.739174 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739134    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7ea7d464-a8e8-4c9e-b437-93784c147707-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.739382 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739196    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.739382 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739222    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-socket-dir\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.739382 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739261    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ea7d464-a8e8-4c9e-b437-93784c147707-system-cni-dir\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.739382 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739299    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mwtc\" (UniqueName: \"kubernetes.io/projected/1f737909-4b95-481b-b092-0816b5be6f34-kube-api-access-5mwtc\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:04.739382 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739323    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjgxb\" (UniqueName: \"kubernetes.io/projected/49ffc9f3-7c5f-48bc-8217-838ac6651979-kube-api-access-rjgxb\") pod \"node-ca-6gkbl\" (UID: \"49ffc9f3-7c5f-48bc-8217-838ac6651979\") " pod="openshift-image-registry/node-ca-6gkbl"
Sep 18 16:05:04.739382 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739356    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-registration-dir\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.739690 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739425    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-device-dir\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.739690 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739456    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ea7d464-a8e8-4c9e-b437-93784c147707-cni-binary-copy\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.739690 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739483    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ea7d464-a8e8-4c9e-b437-93784c147707-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.739690 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739652    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.739987 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739952    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 16:05:04.739987 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.739958    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 16:05:04.740178 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.740158    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6qnc4\""
Sep 18 16:05:04.741384 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.741365    2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:05:04.742216 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.742193    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2bl6"
Sep 18 16:05:04.742368 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.742221    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:05:04.742368 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.742275    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bm6xr\""
Sep 18 16:05:04.742746 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.742727    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 16:05:04.744850 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.744830    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 16:05:04.744937 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.744867    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 16:05:04.745069 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.745053    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kcclj\""
Sep 18 16:05:04.765742 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.765715    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5vkcj"
Sep 18 16:05:04.784557 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.784537    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5vkcj"
Sep 18 16:05:04.826984 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.826960    2546 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 16:05:04.839858 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.839837    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.839967 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.839863    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-socket-dir\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.839967 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.839882    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mwtc\" (UniqueName: \"kubernetes.io/projected/1f737909-4b95-481b-b092-0816b5be6f34-kube-api-access-5mwtc\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:04.839967 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.839909    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-multus-socket-dir-parent\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.839967 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.839931    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-run-netns\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.840181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.839976    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-run-multus-certs\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.840181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.839998    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4fqs\" (UniqueName: \"kubernetes.io/projected/5c9479a0-a3ee-41c8-af8f-4a985ab46319-kube-api-access-f4fqs\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.840181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840001    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.840181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840019    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-run-ovn-kubernetes\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.840181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840045    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-device-dir\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.840181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840068    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-sysconfig\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.840181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840071    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-socket-dir\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.840181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840091    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c9479a0-a3ee-41c8-af8f-4a985ab46319-cni-binary-copy\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.840181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840113    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-device-dir\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.840181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840123    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-etc-kubernetes\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.840181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840155    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-run\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.840181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840173    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-sys-fs\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840194    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-host\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840234    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e19f2090-1ca2-4786-979c-f8ae1f2b5688-ovn-node-metrics-cert\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840279    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b72q4\" (UniqueName: \"kubernetes.io/projected/2023cf66-25c9-441a-b1b9-6892cf0b2746-kube-api-access-b72q4\") pod \"node-resolver-g2bl6\" (UID: \"2023cf66-25c9-441a-b1b9-6892cf0b2746\") " pod="openshift-dns/node-resolver-g2bl6"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840307    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43e780cc-b6d9-456a-9dbc-eedb54a237a2-agent-certs\") pod \"konnectivity-agent-js9rq\" (UID: \"43e780cc-b6d9-456a-9dbc-eedb54a237a2\") " pod="kube-system/konnectivity-agent-js9rq"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840323    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840271    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-sys-fs\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840352    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-cni-bin\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840374    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e19f2090-1ca2-4786-979c-f8ae1f2b5688-ovnkube-script-lib\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840394    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvztn\" (UniqueName: \"kubernetes.io/projected/7ea7d464-a8e8-4c9e-b437-93784c147707-kube-api-access-qvztn\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840415    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43e780cc-b6d9-456a-9dbc-eedb54a237a2-konnectivity-ca\") pod \"konnectivity-agent-js9rq\" (UID: \"43e780cc-b6d9-456a-9dbc-eedb54a237a2\") " pod="kube-system/konnectivity-agent-js9rq"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840436    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-modprobe-d\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840453    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96b69886-eeef-49e1-b8d4-d869f4772457-tmp\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840467    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-systemd-units\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840482    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-run-openvswitch\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840498    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/880072cc-3551-471f-b2f3-17b5a947dc82-iptables-alerter-script\") pod \"iptables-alerter-7xz8m\" (UID: \"880072cc-3551-471f-b2f3-17b5a947dc82\") " pod="openshift-network-operator/iptables-alerter-7xz8m"
Sep 18 16:05:04.840706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840521    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlfxc\" (UniqueName: \"kubernetes.io/projected/880072cc-3551-471f-b2f3-17b5a947dc82-kube-api-access-qlfxc\") pod \"iptables-alerter-7xz8m\" (UID: \"880072cc-3551-471f-b2f3-17b5a947dc82\") " pod="openshift-network-operator/iptables-alerter-7xz8m"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840539    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wlpr\" (UniqueName: \"kubernetes.io/projected/3db26da6-d684-4099-88bd-68e1130faa42-kube-api-access-9wlpr\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840579    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840599    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-var-lib-kubelet\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840633    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/96b69886-eeef-49e1-b8d4-d869f4772457-etc-tuned\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840663    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-slash\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840690    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/880072cc-3551-471f-b2f3-17b5a947dc82-host-slash\") pod \"iptables-alerter-7xz8m\" (UID: \"880072cc-3551-471f-b2f3-17b5a947dc82\") " pod="openshift-network-operator/iptables-alerter-7xz8m"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:04.840729    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840737    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-var-lib-cni-bin\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840798    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/880072cc-3551-471f-b2f3-17b5a947dc82-host-slash\") pod \"iptables-alerter-7xz8m\" (UID: \"880072cc-3551-471f-b2f3-17b5a947dc82\") " pod="openshift-network-operator/iptables-alerter-7xz8m"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:04.840800    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs podName:1f737909-4b95-481b-b092-0816b5be6f34 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:05.340778186 +0000 UTC m=+2.985342125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs") pod "network-metrics-daemon-7h8rh" (UID: "1f737909-4b95-481b-b092-0816b5be6f34") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840837    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-sysctl-conf\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840857    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-kubelet\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840870    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-node-log\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840900    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-log-socket\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840921    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ea7d464-a8e8-4c9e-b437-93784c147707-system-cni-dir\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.841377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840976    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ea7d464-a8e8-4c9e-b437-93784c147707-system-cni-dir\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.840977    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjgxb\" (UniqueName: \"kubernetes.io/projected/49ffc9f3-7c5f-48bc-8217-838ac6651979-kube-api-access-rjgxb\") pod \"node-ca-6gkbl\" (UID: \"49ffc9f3-7c5f-48bc-8217-838ac6651979\") " pod="openshift-image-registry/node-ca-6gkbl"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841024    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-multus-cni-dir\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841050    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8tb\" (UniqueName: \"kubernetes.io/projected/96b69886-eeef-49e1-b8d4-d869f4772457-kube-api-access-jg8tb\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841080    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-registration-dir\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841098    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/880072cc-3551-471f-b2f3-17b5a947dc82-iptables-alerter-script\") pod \"iptables-alerter-7xz8m\" (UID: \"880072cc-3551-471f-b2f3-17b5a947dc82\") " pod="openshift-network-operator/iptables-alerter-7xz8m"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841105    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ea7d464-a8e8-4c9e-b437-93784c147707-cni-binary-copy\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841131    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ea7d464-a8e8-4c9e-b437-93784c147707-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841139    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-registration-dir\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841152    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49ffc9f3-7c5f-48bc-8217-838ac6651979-host\") pod \"node-ca-6gkbl\" (UID: \"49ffc9f3-7c5f-48bc-8217-838ac6651979\") " pod="openshift-image-registry/node-ca-6gkbl"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841168    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-system-cni-dir\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841191    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-var-lib-kubelet\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841210    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49ffc9f3-7c5f-48bc-8217-838ac6651979-host\") pod \"node-ca-6gkbl\" (UID: \"49ffc9f3-7c5f-48bc-8217-838ac6651979\") " pod="openshift-image-registry/node-ca-6gkbl"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841212    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-hostroot\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841271    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-lib-modules\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841299    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7ea7d464-a8e8-4c9e-b437-93784c147707-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841326    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ea7d464-a8e8-4c9e-b437-93784c147707-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.842190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841332    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-cnibin\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841378    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2ctq\" (UniqueName: \"kubernetes.io/projected/e19f2090-1ca2-4786-979c-f8ae1f2b5688-kube-api-access-p2ctq\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841408    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-etc-selinux\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841434    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7ea7d464-a8e8-4c9e-b437-93784c147707-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841460    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-multus-conf-dir\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841518    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3db26da6-d684-4099-88bd-68e1130faa42-etc-selinux\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841552    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-kubernetes\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841578    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-var-lib-cni-multus\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841605    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-sysctl-d\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841626    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7cfa77c5-aa51-41ef-9673-c23c39835582-dbus\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841651    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-run-systemd\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841677    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-etc-openvswitch\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841717    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ea7d464-a8e8-4c9e-b437-93784c147707-cni-binary-copy\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841728    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-cni-netd\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841753    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841768    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7ea7d464-a8e8-4c9e-b437-93784c147707-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.842853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841782    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e19f2090-1ca2-4786-979c-f8ae1f2b5688-ovnkube-config\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841805    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-sys\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841828    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7cfa77c5-aa51-41ef-9673-c23c39835582-kubelet-config\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841853    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-run-netns\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841879    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-var-lib-openvswitch\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841902    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e19f2090-1ca2-4786-979c-f8ae1f2b5688-env-overrides\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841917    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7ea7d464-a8e8-4c9e-b437-93784c147707-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841925    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2023cf66-25c9-441a-b1b9-6892cf0b2746-hosts-file\") pod \"node-resolver-g2bl6\" (UID: \"2023cf66-25c9-441a-b1b9-6892cf0b2746\") " pod="openshift-dns/node-resolver-g2bl6"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841946    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2023cf66-25c9-441a-b1b9-6892cf0b2746-tmp-dir\") pod \"node-resolver-g2bl6\" (UID: \"2023cf66-25c9-441a-b1b9-6892cf0b2746\") " pod="openshift-dns/node-resolver-g2bl6"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.841973    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49ffc9f3-7c5f-48bc-8217-838ac6651979-serviceca\") pod \"node-ca-6gkbl\" (UID: \"49ffc9f3-7c5f-48bc-8217-838ac6651979\") " pod="openshift-image-registry/node-ca-6gkbl"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.842032    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c9479a0-a3ee-41c8-af8f-4a985ab46319-multus-daemon-config\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.842098    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dps4t\" (UniqueName: \"kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t\") pod \"network-check-target-p69sq\" (UID: \"e8acfc8b-7c80-4522-9d84-b26a033bc57e\") " pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.842138    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ea7d464-a8e8-4c9e-b437-93784c147707-cnibin\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.842192    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-run-k8s-cni-cncf-io\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.842203    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ea7d464-a8e8-4c9e-b437-93784c147707-cnibin\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.842251    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-systemd\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.842278    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-run-ovn\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.843423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.842536    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49ffc9f3-7c5f-48bc-8217-838ac6651979-serviceca\") pod \"node-ca-6gkbl\" (UID: \"49ffc9f3-7c5f-48bc-8217-838ac6651979\") " pod="openshift-image-registry/node-ca-6gkbl"
Sep 18 16:05:04.858221 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.858192    2546 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 16:05:04.862945 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.862921    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mwtc\" (UniqueName: \"kubernetes.io/projected/1f737909-4b95-481b-b092-0816b5be6f34-kube-api-access-5mwtc\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:04.863069 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.862945    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjgxb\" (UniqueName: \"kubernetes.io/projected/49ffc9f3-7c5f-48bc-8217-838ac6651979-kube-api-access-rjgxb\") pod \"node-ca-6gkbl\" (UID: \"49ffc9f3-7c5f-48bc-8217-838ac6651979\") " pod="openshift-image-registry/node-ca-6gkbl"
Sep 18 16:05:04.863069 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.863019    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvztn\" (UniqueName: \"kubernetes.io/projected/7ea7d464-a8e8-4c9e-b437-93784c147707-kube-api-access-qvztn\") pod \"multus-additional-cni-plugins-qnmct\" (UID: \"7ea7d464-a8e8-4c9e-b437-93784c147707\") " pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:04.863190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.863072    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlfxc\" (UniqueName: \"kubernetes.io/projected/880072cc-3551-471f-b2f3-17b5a947dc82-kube-api-access-qlfxc\") pod \"iptables-alerter-7xz8m\" (UID: \"880072cc-3551-471f-b2f3-17b5a947dc82\") " pod="openshift-network-operator/iptables-alerter-7xz8m"
Sep 18 16:05:04.863298 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.863274    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wlpr\" (UniqueName: \"kubernetes.io/projected/3db26da6-d684-4099-88bd-68e1130faa42-kube-api-access-9wlpr\") pod \"aws-ebs-csi-driver-node-hrml2\" (UID: \"3db26da6-d684-4099-88bd-68e1130faa42\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:04.865157 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.865112    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal" event={"ID":"257b3b74c0f4140f52501737bffd4b82","Type":"ContainerStarted","Data":"53e8bc082cee604e4754bce3268a54e3608fc341153913f791d526d8ba2d3733"}
Sep 18 16:05:04.867143 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.867122    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-253.ec2.internal" event={"ID":"93cd48393ba29a5c2da55b21c3ff83e3","Type":"ContainerStarted","Data":"390281c65d17d931f66e355cb3f4150606f36ea69566f334b58a3e7749efd1ca"}
Sep 18 16:05:04.942564 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942524    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-var-lib-cni-multus\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.942564 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942563    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-sysctl-d\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.942741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942588    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7cfa77c5-aa51-41ef-9673-c23c39835582-dbus\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:04.942741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942607    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-run-systemd\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.942741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942621    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-etc-openvswitch\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.942741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942619    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-var-lib-cni-multus\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.942741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942637    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-cni-netd\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.942741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942674    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.942741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942695    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-etc-openvswitch\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.942741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942701    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e19f2090-1ca2-4786-979c-f8ae1f2b5688-ovnkube-config\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.942741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942736    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-sys\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.942741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942741    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-cni-netd\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942759    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7cfa77c5-aa51-41ef-9673-c23c39835582-kubelet-config\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942784    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-run-netns\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942829    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7cfa77c5-aa51-41ef-9673-c23c39835582-kubelet-config\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942838    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-var-lib-openvswitch\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942835    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-sysctl-d\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942835    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-sys\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942784    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942881    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-run-netns\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942875    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e19f2090-1ca2-4786-979c-f8ae1f2b5688-env-overrides\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942914    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-var-lib-openvswitch\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942904    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-run-systemd\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942833    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7cfa77c5-aa51-41ef-9673-c23c39835582-dbus\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942933    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2023cf66-25c9-441a-b1b9-6892cf0b2746-hosts-file\") pod \"node-resolver-g2bl6\" (UID: \"2023cf66-25c9-441a-b1b9-6892cf0b2746\") " pod="openshift-dns/node-resolver-g2bl6"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942966    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2023cf66-25c9-441a-b1b9-6892cf0b2746-tmp-dir\") pod \"node-resolver-g2bl6\" (UID: \"2023cf66-25c9-441a-b1b9-6892cf0b2746\") " pod="openshift-dns/node-resolver-g2bl6"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942973    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2023cf66-25c9-441a-b1b9-6892cf0b2746-hosts-file\") pod \"node-resolver-g2bl6\" (UID: \"2023cf66-25c9-441a-b1b9-6892cf0b2746\") " pod="openshift-dns/node-resolver-g2bl6"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.942992    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c9479a0-a3ee-41c8-af8f-4a985ab46319-multus-daemon-config\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.943212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943017    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dps4t\" (UniqueName: \"kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t\") pod \"network-check-target-p69sq\" (UID: \"e8acfc8b-7c80-4522-9d84-b26a033bc57e\") " pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943042    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-run-k8s-cni-cncf-io\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943066    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-systemd\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943087    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-run-ovn\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943112    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-multus-socket-dir-parent\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943133    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-run-netns\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943155    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-run-multus-certs\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943187    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4fqs\" (UniqueName: \"kubernetes.io/projected/5c9479a0-a3ee-41c8-af8f-4a985ab46319-kube-api-access-f4fqs\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943210    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-run-ovn-kubernetes\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943255    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-sysconfig\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943265    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-run-netns\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943296    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c9479a0-a3ee-41c8-af8f-4a985ab46319-cni-binary-copy\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943301    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2023cf66-25c9-441a-b1b9-6892cf0b2746-tmp-dir\") pod \"node-resolver-g2bl6\" (UID: \"2023cf66-25c9-441a-b1b9-6892cf0b2746\") " pod="openshift-dns/node-resolver-g2bl6"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943315    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-systemd\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943320    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-etc-kubernetes\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943318    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e19f2090-1ca2-4786-979c-f8ae1f2b5688-ovnkube-config\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943130    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-run-k8s-cni-cncf-io\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943341    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-run-ovn\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.944007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943362    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-run\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943395    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-multus-socket-dir-parent\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943411    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-etc-kubernetes\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943422    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-sysconfig\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943445    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-host\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943456    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-run\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943465    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e19f2090-1ca2-4786-979c-f8ae1f2b5688-ovn-node-metrics-cert\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943497    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-run-multus-certs\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943504    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-host\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943523    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-run-ovn-kubernetes\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943532    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b72q4\" (UniqueName: \"kubernetes.io/projected/2023cf66-25c9-441a-b1b9-6892cf0b2746-kube-api-access-b72q4\") pod \"node-resolver-g2bl6\" (UID: \"2023cf66-25c9-441a-b1b9-6892cf0b2746\") " pod="openshift-dns/node-resolver-g2bl6"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943598    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5c9479a0-a3ee-41c8-af8f-4a985ab46319-multus-daemon-config\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943649    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43e780cc-b6d9-456a-9dbc-eedb54a237a2-agent-certs\") pod \"konnectivity-agent-js9rq\" (UID: \"43e780cc-b6d9-456a-9dbc-eedb54a237a2\") " pod="kube-system/konnectivity-agent-js9rq"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943691    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943715    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-cni-bin\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943751    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e19f2090-1ca2-4786-979c-f8ae1f2b5688-ovnkube-script-lib\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943792    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-cni-bin\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:04.943836    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:04.944840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943879    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43e780cc-b6d9-456a-9dbc-eedb54a237a2-konnectivity-ca\") pod \"konnectivity-agent-js9rq\" (UID: \"43e780cc-b6d9-456a-9dbc-eedb54a237a2\") " pod="kube-system/konnectivity-agent-js9rq"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943891    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5c9479a0-a3ee-41c8-af8f-4a985ab46319-cni-binary-copy\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943925    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e19f2090-1ca2-4786-979c-f8ae1f2b5688-env-overrides\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943927    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-modprobe-d\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.943967    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96b69886-eeef-49e1-b8d4-d869f4772457-tmp\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:04.944035    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret podName:7cfa77c5-aa51-41ef-9673-c23c39835582 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:05.444014627 +0000 UTC m=+3.088578580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret") pod "global-pull-secret-syncer-pcmn7" (UID: "7cfa77c5-aa51-41ef-9673-c23c39835582") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944098    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-systemd-units\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944126    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-run-openvswitch\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944230    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-systemd-units\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944285    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-run-openvswitch\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944337    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-modprobe-d\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944361    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-var-lib-kubelet\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944384    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/96b69886-eeef-49e1-b8d4-d869f4772457-etc-tuned\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944409    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-slash\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944426    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-var-lib-kubelet\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944436    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-var-lib-cni-bin\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944455    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/43e780cc-b6d9-456a-9dbc-eedb54a237a2-konnectivity-ca\") pod \"konnectivity-agent-js9rq\" (UID: \"43e780cc-b6d9-456a-9dbc-eedb54a237a2\") " pod="kube-system/konnectivity-agent-js9rq"
Sep 18 16:05:04.945579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944459    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e19f2090-1ca2-4786-979c-f8ae1f2b5688-ovnkube-script-lib\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944461    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-sysctl-conf\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944501    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-var-lib-cni-bin\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944515    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-kubelet\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944541    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-node-log\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944537    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-slash\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944564    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-log-socket\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944589    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-host-kubelet\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944587    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-sysctl-conf\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944590    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-multus-cni-dir\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944628    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8tb\" (UniqueName: \"kubernetes.io/projected/96b69886-eeef-49e1-b8d4-d869f4772457-kube-api-access-jg8tb\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944635    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-multus-cni-dir\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944648    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-node-log\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944675    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e19f2090-1ca2-4786-979c-f8ae1f2b5688-log-socket\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944699    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-system-cni-dir\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944724    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-var-lib-kubelet\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944745    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-hostroot\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944767    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-lib-modules\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.946231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944783    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-host-var-lib-kubelet\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944789    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-cnibin\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944817    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-system-cni-dir\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944824    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-hostroot\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944841    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-cnibin\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944855    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2ctq\" (UniqueName: \"kubernetes.io/projected/e19f2090-1ca2-4786-979c-f8ae1f2b5688-kube-api-access-p2ctq\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944891    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-multus-conf-dir\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944895    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-lib-modules\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944915    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-kubernetes\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944945    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5c9479a0-a3ee-41c8-af8f-4a985ab46319-multus-conf-dir\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.944976    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96b69886-eeef-49e1-b8d4-d869f4772457-etc-kubernetes\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.946193    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e19f2090-1ca2-4786-979c-f8ae1f2b5688-ovn-node-metrics-cert\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.946646    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96b69886-eeef-49e1-b8d4-d869f4772457-tmp\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.946685    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/43e780cc-b6d9-456a-9dbc-eedb54a237a2-agent-certs\") pod \"konnectivity-agent-js9rq\" (UID: \"43e780cc-b6d9-456a-9dbc-eedb54a237a2\") " pod="kube-system/konnectivity-agent-js9rq"
Sep 18 16:05:04.947001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.946990    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/96b69886-eeef-49e1-b8d4-d869f4772457-etc-tuned\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.954816 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:04.954796    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:05:04.954816 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:04.954818    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:05:04.954966 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:04.954827    2546 projected.go:196] Error preparing data for projected volume kube-api-access-dps4t for pod openshift-network-diagnostics/network-check-target-p69sq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:04.954966 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:04.954884    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t podName:e8acfc8b-7c80-4522-9d84-b26a033bc57e nodeName:}" failed. No retries permitted until 2026-09-18 16:05:05.454866658 +0000 UTC m=+3.099430602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dps4t" (UniqueName: "kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t") pod "network-check-target-p69sq" (UID: "e8acfc8b-7c80-4522-9d84-b26a033bc57e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:04.956766 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.956742    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4fqs\" (UniqueName: \"kubernetes.io/projected/5c9479a0-a3ee-41c8-af8f-4a985ab46319-kube-api-access-f4fqs\") pod \"multus-xxtjw\" (UID: \"5c9479a0-a3ee-41c8-af8f-4a985ab46319\") " pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:04.957372 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.957355    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b72q4\" (UniqueName: \"kubernetes.io/projected/2023cf66-25c9-441a-b1b9-6892cf0b2746-kube-api-access-b72q4\") pod \"node-resolver-g2bl6\" (UID: \"2023cf66-25c9-441a-b1b9-6892cf0b2746\") " pod="openshift-dns/node-resolver-g2bl6"
Sep 18 16:05:04.961893 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.961840    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8tb\" (UniqueName: \"kubernetes.io/projected/96b69886-eeef-49e1-b8d4-d869f4772457-kube-api-access-jg8tb\") pod \"tuned-jxv46\" (UID: \"96b69886-eeef-49e1-b8d4-d869f4772457\") " pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:04.962319 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:04.962299    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2ctq\" (UniqueName: \"kubernetes.io/projected/e19f2090-1ca2-4786-979c-f8ae1f2b5688-kube-api-access-p2ctq\") pod \"ovnkube-node-dnn6j\" (UID: \"e19f2090-1ca2-4786-979c-f8ae1f2b5688\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:05.030143 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:05.030063    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6gkbl"
Sep 18 16:05:05.036662 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:05.036634    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7xz8m"
Sep 18 16:05:05.044329 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:05.044308    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2"
Sep 18 16:05:05.049954 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:05.049934    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qnmct"
Sep 18 16:05:05.056649 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:05.056630    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xxtjw"
Sep 18 16:05:05.065063 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:05.065045    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:05.069599 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:05.069581    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-js9rq"
Sep 18 16:05:05.076130 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:05.076115    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jxv46"
Sep 18 16:05:05.081661 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:05.081642    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2bl6"
Sep 18 16:05:07.083517 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.083441    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:07.083667 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:07.083605    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:07.083736 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:07.083677    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs podName:1f737909-4b95-481b-b092-0816b5be6f34 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:08.083656223 +0000 UTC m=+3.992436566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs") pod "network-metrics-daemon-7h8rh" (UID: "1f737909-4b95-481b-b092-0816b5be6f34") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:07.125600 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:07.125568    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea7d464_a8e8_4c9e_b437_93784c147707.slice/crio-1dac48ae7528377005097fe1e4c572e2ff089c71f7ec1b031ce1dab9355e4af1 WatchSource:0}: Error finding container 1dac48ae7528377005097fe1e4c572e2ff089c71f7ec1b031ce1dab9355e4af1: Status 404 returned error can't find the container with id 1dac48ae7528377005097fe1e4c572e2ff089c71f7ec1b031ce1dab9355e4af1
Sep 18 16:05:07.128598 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:07.128458    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db26da6_d684_4099_88bd_68e1130faa42.slice/crio-b2b497a09e0956a20d1d5101f06fb0b10f381600ba7098f38d9f520a9753b39c WatchSource:0}: Error finding container b2b497a09e0956a20d1d5101f06fb0b10f381600ba7098f38d9f520a9753b39c: Status 404 returned error can't find the container with id b2b497a09e0956a20d1d5101f06fb0b10f381600ba7098f38d9f520a9753b39c
Sep 18 16:05:07.130600 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:07.130571    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880072cc_3551_471f_b2f3_17b5a947dc82.slice/crio-0d44ce1ca93aded4fe1b1aeeaa4e2a77bbbf216863aa178e561fc8a580960dea WatchSource:0}: Error finding container 0d44ce1ca93aded4fe1b1aeeaa4e2a77bbbf216863aa178e561fc8a580960dea: Status 404 returned error can't find the container with id 0d44ce1ca93aded4fe1b1aeeaa4e2a77bbbf216863aa178e561fc8a580960dea
Sep 18 16:05:07.131540 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:07.131520    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b69886_eeef_49e1_b8d4_d869f4772457.slice/crio-e3ea0212f40decec0b9f6b32aca4e1b14d34bc6b3306f4080f14b5a29b8de53d WatchSource:0}: Error finding container e3ea0212f40decec0b9f6b32aca4e1b14d34bc6b3306f4080f14b5a29b8de53d: Status 404 returned error can't find the container with id e3ea0212f40decec0b9f6b32aca4e1b14d34bc6b3306f4080f14b5a29b8de53d
Sep 18 16:05:07.132490 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:07.132469    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9479a0_a3ee_41c8_af8f_4a985ab46319.slice/crio-b83e5a8d60a6f7d9c10a91508f696fa7dee27be0076c89c54476a6a24f9d1f31 WatchSource:0}: Error finding container b83e5a8d60a6f7d9c10a91508f696fa7dee27be0076c89c54476a6a24f9d1f31: Status 404 returned error can't find the container with id b83e5a8d60a6f7d9c10a91508f696fa7dee27be0076c89c54476a6a24f9d1f31
Sep 18 16:05:07.133782 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:07.133762    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19f2090_1ca2_4786_979c_f8ae1f2b5688.slice/crio-3e1e47c9b4abd472afcae374c9fdaabe4db211d2601c6e77a670c51c11dd183b WatchSource:0}: Error finding container 3e1e47c9b4abd472afcae374c9fdaabe4db211d2601c6e77a670c51c11dd183b: Status 404 returned error can't find the container with id 3e1e47c9b4abd472afcae374c9fdaabe4db211d2601c6e77a670c51c11dd183b
Sep 18 16:05:07.157803 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:07.157775    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2023cf66_25c9_441a_b1b9_6892cf0b2746.slice/crio-8da45ae45ace915dfe19103ed8372662d85af15be33fa1cd9491325b4acb0e9c WatchSource:0}: Error finding container 8da45ae45ace915dfe19103ed8372662d85af15be33fa1cd9491325b4acb0e9c: Status 404 returned error can't find the container with id 8da45ae45ace915dfe19103ed8372662d85af15be33fa1cd9491325b4acb0e9c
Sep 18 16:05:07.158271 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:07.158225    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ffc9f3_7c5f_48bc_8217_838ac6651979.slice/crio-4e3ea2869449e3ccffee9f195beb8a0f1ad45701b1c84094063b805f430b4aca WatchSource:0}: Error finding container 4e3ea2869449e3ccffee9f195beb8a0f1ad45701b1c84094063b805f430b4aca: Status 404 returned error can't find the container with id 4e3ea2869449e3ccffee9f195beb8a0f1ad45701b1c84094063b805f430b4aca
Sep 18 16:05:07.159006 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:07.158983    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e780cc_b6d9_456a_9dbc_eedb54a237a2.slice/crio-f6014ddeffe2b71c93eb394b37e7a4b462908f5761481bc83784d40b7b029843 WatchSource:0}: Error finding container f6014ddeffe2b71c93eb394b37e7a4b462908f5761481bc83784d40b7b029843: Status 404 returned error can't find the container with id f6014ddeffe2b71c93eb394b37e7a4b462908f5761481bc83784d40b7b029843
Sep 18 16:05:07.184606 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.184546    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:07.184693 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:07.184653    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:07.184729 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:07.184705    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret podName:7cfa77c5-aa51-41ef-9673-c23c39835582 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:08.18469174 +0000 UTC m=+4.093472070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret") pod "global-pull-secret-syncer-pcmn7" (UID: "7cfa77c5-aa51-41ef-9673-c23c39835582") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:07.285905 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.285875    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dps4t\" (UniqueName: \"kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t\") pod \"network-check-target-p69sq\" (UID: \"e8acfc8b-7c80-4522-9d84-b26a033bc57e\") " pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:07.286072 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:07.286051    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:05:07.286143 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:07.286082    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:05:07.286143 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:07.286096    2546 projected.go:196] Error preparing data for projected volume kube-api-access-dps4t for pod openshift-network-diagnostics/network-check-target-p69sq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:07.286220 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:07.286151    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t podName:e8acfc8b-7c80-4522-9d84-b26a033bc57e nodeName:}" failed. No retries permitted until 2026-09-18 16:05:08.286135892 +0000 UTC m=+4.194916221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dps4t" (UniqueName: "kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t") pod "network-check-target-p69sq" (UID: "e8acfc8b-7c80-4522-9d84-b26a033bc57e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:07.524929 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.524833    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:00:06 +0000 UTC" deadline="2028-03-14 13:59:46.842929 +0000 UTC"
Sep 18 16:05:07.524929 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.524873    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13029h54m39.318060462s"
Sep 18 16:05:07.596270 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.596240    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:07.596437 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:07.596363    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:07.596506 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.596437    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:07.596560 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:07.596507    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:07.607712 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.607637    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6gkbl" event={"ID":"49ffc9f3-7c5f-48bc-8217-838ac6651979","Type":"ContainerStarted","Data":"4e3ea2869449e3ccffee9f195beb8a0f1ad45701b1c84094063b805f430b4aca"}
Sep 18 16:05:07.610748 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.610671    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2bl6" event={"ID":"2023cf66-25c9-441a-b1b9-6892cf0b2746","Type":"ContainerStarted","Data":"8da45ae45ace915dfe19103ed8372662d85af15be33fa1cd9491325b4acb0e9c"}
Sep 18 16:05:07.613214 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.613090    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j" event={"ID":"e19f2090-1ca2-4786-979c-f8ae1f2b5688","Type":"ContainerStarted","Data":"3e1e47c9b4abd472afcae374c9fdaabe4db211d2601c6e77a670c51c11dd183b"}
Sep 18 16:05:07.616189 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.616084    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7xz8m" event={"ID":"880072cc-3551-471f-b2f3-17b5a947dc82","Type":"ContainerStarted","Data":"0d44ce1ca93aded4fe1b1aeeaa4e2a77bbbf216863aa178e561fc8a580960dea"}
Sep 18 16:05:07.617796 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.617771    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xxtjw" event={"ID":"5c9479a0-a3ee-41c8-af8f-4a985ab46319","Type":"ContainerStarted","Data":"b83e5a8d60a6f7d9c10a91508f696fa7dee27be0076c89c54476a6a24f9d1f31"}
Sep 18 16:05:07.621124 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.621102    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2" event={"ID":"3db26da6-d684-4099-88bd-68e1130faa42","Type":"ContainerStarted","Data":"b2b497a09e0956a20d1d5101f06fb0b10f381600ba7098f38d9f520a9753b39c"}
Sep 18 16:05:07.635017 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.634965    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnmct" event={"ID":"7ea7d464-a8e8-4c9e-b437-93784c147707","Type":"ContainerStarted","Data":"1dac48ae7528377005097fe1e4c572e2ff089c71f7ec1b031ce1dab9355e4af1"}
Sep 18 16:05:07.647139 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.646607    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-253.ec2.internal" event={"ID":"93cd48393ba29a5c2da55b21c3ff83e3","Type":"ContainerStarted","Data":"1c085c6659f86748b081dc806101fd27cc21e4e434befd2e90678aa8890efa00"}
Sep 18 16:05:07.654495 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.654473    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jxv46" event={"ID":"96b69886-eeef-49e1-b8d4-d869f4772457","Type":"ContainerStarted","Data":"e3ea0212f40decec0b9f6b32aca4e1b14d34bc6b3306f4080f14b5a29b8de53d"}
Sep 18 16:05:07.660761 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:07.660709    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-js9rq" event={"ID":"43e780cc-b6d9-456a-9dbc-eedb54a237a2","Type":"ContainerStarted","Data":"f6014ddeffe2b71c93eb394b37e7a4b462908f5761481bc83784d40b7b029843"}
Sep 18 16:05:08.092317 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:08.092278    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:08.092521 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:08.092503    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:08.092615 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:08.092569    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs podName:1f737909-4b95-481b-b092-0816b5be6f34 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:10.09255201 +0000 UTC m=+6.001332363 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs") pod "network-metrics-daemon-7h8rh" (UID: "1f737909-4b95-481b-b092-0816b5be6f34") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:08.194470 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:08.194428    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:08.203518 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:08.203481    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:08.203663 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:08.203610    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret podName:7cfa77c5-aa51-41ef-9673-c23c39835582 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:10.203581476 +0000 UTC m=+6.112361827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret") pod "global-pull-secret-syncer-pcmn7" (UID: "7cfa77c5-aa51-41ef-9673-c23c39835582") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:08.296236 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:08.296197    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dps4t\" (UniqueName: \"kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t\") pod \"network-check-target-p69sq\" (UID: \"e8acfc8b-7c80-4522-9d84-b26a033bc57e\") " pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:08.296419 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:08.296353    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:05:08.296419 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:08.296376    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:05:08.296419 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:08.296387    2546 projected.go:196] Error preparing data for projected volume kube-api-access-dps4t for pod openshift-network-diagnostics/network-check-target-p69sq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:08.296567 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:08.296485    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t podName:e8acfc8b-7c80-4522-9d84-b26a033bc57e nodeName:}" failed. No retries permitted until 2026-09-18 16:05:10.296465245 +0000 UTC m=+6.205245593 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dps4t" (UniqueName: "kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t") pod "network-check-target-p69sq" (UID: "e8acfc8b-7c80-4522-9d84-b26a033bc57e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:08.525475 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:08.525410    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:00:06 +0000 UTC" deadline="2028-03-10 03:43:21.255023947 +0000 UTC"
Sep 18 16:05:08.525475 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:08.525473    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12923h38m12.729555556s"
Sep 18 16:05:08.598853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:08.596670    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:08.598853 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:08.596821    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:09.596761 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:09.596547    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:09.597228 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:09.596561    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:09.597228 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:09.596868    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:09.597228 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:09.596950    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:10.117843 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:10.117732    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:10.118004 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:10.117931    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:10.118086 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:10.118038    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs podName:1f737909-4b95-481b-b092-0816b5be6f34 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:14.118002371 +0000 UTC m=+10.026782708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs") pod "network-metrics-daemon-7h8rh" (UID: "1f737909-4b95-481b-b092-0816b5be6f34") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:10.219126 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:10.219089    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:10.219287 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:10.219255    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:10.219358 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:10.219323    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret podName:7cfa77c5-aa51-41ef-9673-c23c39835582 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:14.219303582 +0000 UTC m=+10.128083919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret") pod "global-pull-secret-syncer-pcmn7" (UID: "7cfa77c5-aa51-41ef-9673-c23c39835582") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:10.320982 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:10.320941    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dps4t\" (UniqueName: \"kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t\") pod \"network-check-target-p69sq\" (UID: \"e8acfc8b-7c80-4522-9d84-b26a033bc57e\") " pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:10.321182 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:10.321154    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:05:10.321182 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:10.321174    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:05:10.321302 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:10.321186    2546 projected.go:196] Error preparing data for projected volume kube-api-access-dps4t for pod openshift-network-diagnostics/network-check-target-p69sq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:10.321302 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:10.321244    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t podName:e8acfc8b-7c80-4522-9d84-b26a033bc57e nodeName:}" failed. No retries permitted until 2026-09-18 16:05:14.321227358 +0000 UTC m=+10.230007694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dps4t" (UniqueName: "kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t") pod "network-check-target-p69sq" (UID: "e8acfc8b-7c80-4522-9d84-b26a033bc57e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:10.598830 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:10.598795    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:10.599319 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:10.598944    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:10.675190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:10.675153    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal" event={"ID":"257b3b74c0f4140f52501737bffd4b82","Type":"ContainerStarted","Data":"a19e9b89f15bb166ac7bbecb6ee13eb4c780fa83be44bb54f9c8b36541a5669a"}
Sep 18 16:05:10.691665 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:10.691602    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-253.ec2.internal" podStartSLOduration=5.691579614 podStartE2EDuration="5.691579614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:07.664501822 +0000 UTC m=+3.573282181" watchObservedRunningTime="2026-09-18 16:05:10.691579614 +0000 UTC m=+6.600359966"
Sep 18 16:05:11.596620 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:11.596586    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:11.596798 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:11.596717    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:11.596972 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:11.596952    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:11.597100 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:11.597078    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:12.597136 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:12.597048    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:12.597573 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:12.597194    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:12.687559 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:12.687517    2546 generic.go:356] "Generic (PLEG): container finished" podID="257b3b74c0f4140f52501737bffd4b82" containerID="a19e9b89f15bb166ac7bbecb6ee13eb4c780fa83be44bb54f9c8b36541a5669a" exitCode=0
Sep 18 16:05:12.687758 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:12.687602    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal" event={"ID":"257b3b74c0f4140f52501737bffd4b82","Type":"ContainerDied","Data":"a19e9b89f15bb166ac7bbecb6ee13eb4c780fa83be44bb54f9c8b36541a5669a"}
Sep 18 16:05:13.597172 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:13.597066    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:13.597172 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:13.597115    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:13.597687 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:13.597204    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:13.597687 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:13.597348    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:14.159418 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:14.159377    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:14.159602 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:14.159532    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:14.159602 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:14.159599    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs podName:1f737909-4b95-481b-b092-0816b5be6f34 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:22.159579105 +0000 UTC m=+18.068359439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs") pod "network-metrics-daemon-7h8rh" (UID: "1f737909-4b95-481b-b092-0816b5be6f34") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:14.260589 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:14.260535    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:14.260762 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:14.260712    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:14.260820 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:14.260783    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret podName:7cfa77c5-aa51-41ef-9673-c23c39835582 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:22.260765993 +0000 UTC m=+18.169546330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret") pod "global-pull-secret-syncer-pcmn7" (UID: "7cfa77c5-aa51-41ef-9673-c23c39835582") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:14.361711 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:14.361670    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dps4t\" (UniqueName: \"kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t\") pod \"network-check-target-p69sq\" (UID: \"e8acfc8b-7c80-4522-9d84-b26a033bc57e\") " pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:14.362354 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:14.361887    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:05:14.362354 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:14.361910    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:05:14.362354 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:14.361930    2546 projected.go:196] Error preparing data for projected volume kube-api-access-dps4t for pod openshift-network-diagnostics/network-check-target-p69sq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:14.362354 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:14.361992    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t podName:e8acfc8b-7c80-4522-9d84-b26a033bc57e nodeName:}" failed. No retries permitted until 2026-09-18 16:05:22.361974996 +0000 UTC m=+18.270755365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dps4t" (UniqueName: "kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t") pod "network-check-target-p69sq" (UID: "e8acfc8b-7c80-4522-9d84-b26a033bc57e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:14.600292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:14.600261    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:14.600732 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:14.600393    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:15.597441 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:15.596963    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:15.597441 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:15.596987    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:15.597441 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:15.597117    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:15.597441 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:15.597222    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:16.597117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:16.597082    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:16.597543 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:16.597233    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:17.597062 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:17.596895    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:17.597062 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:17.596930    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:17.597062 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:17.596993    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:17.597533 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:17.597141    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:18.596265 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:18.596231    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:18.596433 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:18.596370    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:19.597141 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:19.597110    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:19.597545 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:19.597112    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:19.597545 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:19.597228    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:19.597545 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:19.597312    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:20.599617 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:20.599591    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:20.600005 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:20.599751    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:21.596433 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:21.596404    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:21.596701 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:21.596405    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:21.596701 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:21.596524    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:21.596701 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:21.596607    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:22.220534 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:22.220499    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:22.220943 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:22.220671    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:22.220943 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:22.220743    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs podName:1f737909-4b95-481b-b092-0816b5be6f34 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:38.220723535 +0000 UTC m=+34.129503873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs") pod "network-metrics-daemon-7h8rh" (UID: "1f737909-4b95-481b-b092-0816b5be6f34") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:05:22.320933 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:22.320896    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:22.321098 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:22.321070    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:22.321154 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:22.321139    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret podName:7cfa77c5-aa51-41ef-9673-c23c39835582 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:38.321120273 +0000 UTC m=+34.229900622 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret") pod "global-pull-secret-syncer-pcmn7" (UID: "7cfa77c5-aa51-41ef-9673-c23c39835582") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:05:22.422159 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:22.422128    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dps4t\" (UniqueName: \"kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t\") pod \"network-check-target-p69sq\" (UID: \"e8acfc8b-7c80-4522-9d84-b26a033bc57e\") " pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:22.422341 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:22.422308    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:05:22.422341 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:22.422333    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:05:22.422445 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:22.422345    2546 projected.go:196] Error preparing data for projected volume kube-api-access-dps4t for pod openshift-network-diagnostics/network-check-target-p69sq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:22.422445 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:22.422408    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t podName:e8acfc8b-7c80-4522-9d84-b26a033bc57e nodeName:}" failed. No retries permitted until 2026-09-18 16:05:38.422388094 +0000 UTC m=+34.331168429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dps4t" (UniqueName: "kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t") pod "network-check-target-p69sq" (UID: "e8acfc8b-7c80-4522-9d84-b26a033bc57e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:05:22.597197 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:22.597115    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:22.597362 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:22.597252    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:23.596331 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:23.596302    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:23.596790 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:23.596423    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:23.596790 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:23.596472    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:23.596790 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:23.596564    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:24.597958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:24.597929    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:24.598277 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:24.598065    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:25.596684 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.596290    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:25.596684 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.596334    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:25.596897 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:25.596718    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:25.596897 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:25.596775    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:25.712550 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.712514    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal" event={"ID":"257b3b74c0f4140f52501737bffd4b82","Type":"ContainerStarted","Data":"5f9e380751564a9e63f85e834e19d5d2c69f6ca1df5abf506c700b22947a4bf7"}
Sep 18 16:05:25.714398 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.714209    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jxv46" event={"ID":"96b69886-eeef-49e1-b8d4-d869f4772457","Type":"ContainerStarted","Data":"0333488504e6ff36b50801b870b5a9fb163d1f8580138e83a65516e441ab350a"}
Sep 18 16:05:25.716214 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.716187    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-js9rq" event={"ID":"43e780cc-b6d9-456a-9dbc-eedb54a237a2","Type":"ContainerStarted","Data":"f016942632474d8c9d75d3fcfd9294f3f698061ca85e25adbda89817c66914aa"}
Sep 18 16:05:25.717822 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.717798    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6gkbl" event={"ID":"49ffc9f3-7c5f-48bc-8217-838ac6651979","Type":"ContainerStarted","Data":"4b433421affffe1efbc7edd9a5f2528fb800af632b2bded83277173af620a026"}
Sep 18 16:05:25.719460 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.719440    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2bl6" event={"ID":"2023cf66-25c9-441a-b1b9-6892cf0b2746","Type":"ContainerStarted","Data":"8c2b8087e9923df404c40d9f952d9b3cc2e7c55396d8e2798e31fbb2a3ba0fb7"}
Sep 18 16:05:25.722935 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.722910    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j" event={"ID":"e19f2090-1ca2-4786-979c-f8ae1f2b5688","Type":"ContainerStarted","Data":"4d0087fd0ae16bbed7ee74937a04461304c732e33c4686051f1152f000e876a6"}
Sep 18 16:05:25.723050 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.722982    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j" event={"ID":"e19f2090-1ca2-4786-979c-f8ae1f2b5688","Type":"ContainerStarted","Data":"8bf34e03a0176911fbc5b797a4dcac98acd9a5ecb95f193019f1c91970db060a"}
Sep 18 16:05:25.723050 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.723045    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j" event={"ID":"e19f2090-1ca2-4786-979c-f8ae1f2b5688","Type":"ContainerStarted","Data":"df326104569fe316e4c15790c03b4d7a61d584bfb45e1dbde6fbd620f497ad19"}
Sep 18 16:05:25.723167 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.723093    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j" event={"ID":"e19f2090-1ca2-4786-979c-f8ae1f2b5688","Type":"ContainerStarted","Data":"e142ad739d6c90c52c3aafcea64d32708ec91ef54837b25c74f77841b45658b0"}
Sep 18 16:05:25.723167 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.723139    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j" event={"ID":"e19f2090-1ca2-4786-979c-f8ae1f2b5688","Type":"ContainerStarted","Data":"d13faee3eae923cbc5aee2e63a5deb242524b0f013e5372c530cc1e9fbff07f1"}
Sep 18 16:05:25.723258 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.723184    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j" event={"ID":"e19f2090-1ca2-4786-979c-f8ae1f2b5688","Type":"ContainerStarted","Data":"be3ffbb111480ef70b0084671623b15a03f420e5ebc9913b7e49844430eefed7"}
Sep 18 16:05:25.724606 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.724576    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xxtjw" event={"ID":"5c9479a0-a3ee-41c8-af8f-4a985ab46319","Type":"ContainerStarted","Data":"dc7217e60f2cef385dd0e12b479597d9176b599580335efb4bfb58e78e7ece70"}
Sep 18 16:05:25.726312 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.726288    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2" event={"ID":"3db26da6-d684-4099-88bd-68e1130faa42","Type":"ContainerStarted","Data":"4132768c20028dda98ae8165acf856b725ac43fd2811a6a9fa78b5dd6882ff7d"}
Sep 18 16:05:25.727007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.726966    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-253.ec2.internal" podStartSLOduration=20.726950316 podStartE2EDuration="20.726950316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:25.72691957 +0000 UTC m=+21.635699921" watchObservedRunningTime="2026-09-18 16:05:25.726950316 +0000 UTC m=+21.635730660"
Sep 18 16:05:25.731282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.731256    2546 generic.go:356] "Generic (PLEG): container finished" podID="7ea7d464-a8e8-4c9e-b437-93784c147707" containerID="9f5bb2a89a01ac87cda025d9470954835e46550fae6f3815d7f8d48df57dba74" exitCode=0
Sep 18 16:05:25.731372 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.731300    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnmct" event={"ID":"7ea7d464-a8e8-4c9e-b437-93784c147707","Type":"ContainerDied","Data":"9f5bb2a89a01ac87cda025d9470954835e46550fae6f3815d7f8d48df57dba74"}
Sep 18 16:05:25.742371 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.742336    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6gkbl" podStartSLOduration=4.223627707 podStartE2EDuration="21.742324132s" totalImagesPullingTime="17.518696425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:25.742254172 +0000 UTC m=+21.651034523" watchObservedRunningTime="2026-09-18 16:05:25.742324132 +0000 UTC m=+21.651104482"
Sep 18 16:05:25.776485 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.776441    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xxtjw" podStartSLOduration=4.183502637 podStartE2EDuration="21.776423543s" totalImagesPullingTime="17.592920906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:25.759099599 +0000 UTC m=+21.667879973" watchObservedRunningTime="2026-09-18 16:05:25.776423543 +0000 UTC m=+21.685203897"
Sep 18 16:05:25.776703 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.776670    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-js9rq" podStartSLOduration=4.236003077 podStartE2EDuration="21.776660479s" totalImagesPullingTime="17.540657402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:25.775955109 +0000 UTC m=+21.684735485" watchObservedRunningTime="2026-09-18 16:05:25.776660479 +0000 UTC m=+21.685440829"
Sep 18 16:05:25.794538 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.794505    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jxv46" podStartSLOduration=4.22590208 podStartE2EDuration="21.794492661s" totalImagesPullingTime="17.568590581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:25.794051028 +0000 UTC m=+21.702831396" watchObservedRunningTime="2026-09-18 16:05:25.794492661 +0000 UTC m=+21.703273011"
Sep 18 16:05:25.809424 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:25.809393    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g2bl6" podStartSLOduration=4.290710336 podStartE2EDuration="21.809382558s" totalImagesPullingTime="17.518672222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:25.809328656 +0000 UTC m=+21.718109006" watchObservedRunningTime="2026-09-18 16:05:25.809382558 +0000 UTC m=+21.718162908"
Sep 18 16:05:26.016806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:26.016787    2546 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 16:05:26.528732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:26.528559    2546 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T16:05:26.01680319Z","UUID":"75e2eb61-ba6f-4cfd-8932-815da0d991da","Handler":null,"Name":"","Endpoint":""}
Sep 18 16:05:26.530149 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:26.530124    2546 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 16:05:26.530312 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:26.530157    2546 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 16:05:26.596455 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:26.596429    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:26.596608 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:26.596567    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:26.651520 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:26.651143    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-js9rq"
Sep 18 16:05:26.651994 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:26.651976    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-js9rq"
Sep 18 16:05:26.735829 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:26.735789    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7xz8m" event={"ID":"880072cc-3551-471f-b2f3-17b5a947dc82","Type":"ContainerStarted","Data":"5fb5908be1c36e5db5d78f7fe474eeb2d5654fb0df26fb5530789aa3f2f191b4"}
Sep 18 16:05:26.738300 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:26.738272    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2" event={"ID":"3db26da6-d684-4099-88bd-68e1130faa42","Type":"ContainerStarted","Data":"2c4a4b4ba3b8362f796fd7c214c16220b41f2176ce08fdd9a5d51ab903918647"}
Sep 18 16:05:26.738864 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:26.738840    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-js9rq"
Sep 18 16:05:26.739137 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:26.739121    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-js9rq"
Sep 18 16:05:26.753402 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:26.753360    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7xz8m" podStartSLOduration=5.183922141 podStartE2EDuration="22.753347408s" totalImagesPullingTime="17.569425267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:26.752887606 +0000 UTC m=+22.661667983" watchObservedRunningTime="2026-09-18 16:05:26.753347408 +0000 UTC m=+22.662127759"
Sep 18 16:05:27.596653 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:27.596618    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:27.596748 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:27.596726    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:27.596859 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:27.596838    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:27.596971 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:27.596949    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:27.742985 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:27.742892    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2" event={"ID":"3db26da6-d684-4099-88bd-68e1130faa42","Type":"ContainerStarted","Data":"e784b10d3f11e093986bce3fcc3b34f6716488a89f26c50a58ac2a402209dd2c"}
Sep 18 16:05:27.746846 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:27.746802    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j" event={"ID":"e19f2090-1ca2-4786-979c-f8ae1f2b5688","Type":"ContainerStarted","Data":"5a99fa720ae37b5e66de8f25d41045126af7b8c2b1f8572e250a1d58b904138a"}
Sep 18 16:05:27.763298 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:27.763253    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hrml2" podStartSLOduration=4.339963885 podStartE2EDuration="23.763238811s" totalImagesPullingTime="19.423274926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:27.763178685 +0000 UTC m=+23.671959065" watchObservedRunningTime="2026-09-18 16:05:27.763238811 +0000 UTC m=+23.672019159"
Sep 18 16:05:28.597092 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:28.597056    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:28.597365 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:28.597228    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:29.596528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:29.596498    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:29.596950 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:29.596505    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:29.596950 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:29.596599    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:29.596950 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:29.596691    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:30.596995 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:30.596840    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:30.597625 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:30.597111    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:30.756124 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:30.756100    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j" event={"ID":"e19f2090-1ca2-4786-979c-f8ae1f2b5688","Type":"ContainerStarted","Data":"ac8d0924647d3c95aa3f658583fa414500f24337919408b5d5ad33605e271380"}
Sep 18 16:05:30.756427 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:30.756388    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:30.756601 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:30.756458    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:30.756601 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:30.756505    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:30.758094 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:30.758064    2546 generic.go:356] "Generic (PLEG): container finished" podID="7ea7d464-a8e8-4c9e-b437-93784c147707" containerID="86bdd10eebc2bc91f7221d698d2bbe579858115b2a51dfc91a3be578c72bd851" exitCode=0
Sep 18 16:05:30.758179 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:30.758100    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnmct" event={"ID":"7ea7d464-a8e8-4c9e-b437-93784c147707","Type":"ContainerDied","Data":"86bdd10eebc2bc91f7221d698d2bbe579858115b2a51dfc91a3be578c72bd851"}
Sep 18 16:05:30.772339 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:30.772321    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:30.772545 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:30.772532    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:05:30.784571 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:30.784534    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j" podStartSLOduration=9.101832854 podStartE2EDuration="26.784523852s" totalImagesPullingTime="17.682690998s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:30.784234404 +0000 UTC m=+26.693014754" watchObservedRunningTime="2026-09-18 16:05:30.784523852 +0000 UTC m=+26.693304203"
Sep 18 16:05:31.597056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:31.596872    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:31.597442 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:31.596872    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:31.597442 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:31.597127    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:31.597442 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:31.597191    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:31.599892 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:31.599871    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pcmn7"]
Sep 18 16:05:31.603524 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:31.603501    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p69sq"]
Sep 18 16:05:31.604148 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:31.604130    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7h8rh"]
Sep 18 16:05:31.604229 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:31.604216    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:31.604310 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:31.604291    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:31.759828 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:31.759805    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:31.759942 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:31.759812    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:31.759942 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:31.759897    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:31.760040 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:31.759944    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:32.763814 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:32.763786    2546 generic.go:356] "Generic (PLEG): container finished" podID="7ea7d464-a8e8-4c9e-b437-93784c147707" containerID="1c753088699c4f2e1725a5594e1153f8d9e5525b614797d3ef0a1d9f0dc830ce" exitCode=0
Sep 18 16:05:32.764200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:32.763869    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnmct" event={"ID":"7ea7d464-a8e8-4c9e-b437-93784c147707","Type":"ContainerDied","Data":"1c753088699c4f2e1725a5594e1153f8d9e5525b614797d3ef0a1d9f0dc830ce"}
Sep 18 16:05:33.597039 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:33.596998    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:33.597210 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:33.597053    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:33.597210 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:33.597102    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:33.597210 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:33.597115    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:33.597210 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:33.597174    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:33.597392 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:33.597281    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:34.769297 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:34.769267    2546 generic.go:356] "Generic (PLEG): container finished" podID="7ea7d464-a8e8-4c9e-b437-93784c147707" containerID="97b62dca219964dc31c87a4e8319757f7ccadff7a874836a9fcbf3f416a75d46" exitCode=0
Sep 18 16:05:34.769297 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:34.769298    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnmct" event={"ID":"7ea7d464-a8e8-4c9e-b437-93784c147707","Type":"ContainerDied","Data":"97b62dca219964dc31c87a4e8319757f7ccadff7a874836a9fcbf3f416a75d46"}
Sep 18 16:05:35.596780 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:35.596748    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:35.596932 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:35.596848    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:35.596932 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:35.596858    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:35.596932 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:35.596858    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pcmn7" podUID="7cfa77c5-aa51-41ef-9673-c23c39835582"
Sep 18 16:05:35.596932 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:35.596928    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7h8rh" podUID="1f737909-4b95-481b-b092-0816b5be6f34"
Sep 18 16:05:35.597075 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:35.596988    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p69sq" podUID="e8acfc8b-7c80-4522-9d84-b26a033bc57e"
Sep 18 16:05:37.372919 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.372719    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-253.ec2.internal" event="NodeReady"
Sep 18 16:05:37.373348 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.373036    2546 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 16:05:37.417202 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.417176    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c7lb2"]
Sep 18 16:05:37.419929 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.419907    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:37.419929 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.419932    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r7pf8"]
Sep 18 16:05:37.422192 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.422173    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 16:05:37.422302 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.422179    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bgqnj\""
Sep 18 16:05:37.422302 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.422179    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 16:05:37.422673 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.422651    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r7pf8"
Sep 18 16:05:37.424601 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.424586    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 16:05:37.425042 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.425008    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-69r9k\""
Sep 18 16:05:37.425042 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.425040    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 16:05:37.425300 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.425283    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 16:05:37.429558 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.429539    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c7lb2"]
Sep 18 16:05:37.435207 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.435178    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r7pf8"]
Sep 18 16:05:37.536534 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.536464    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:37.536534 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.536495    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad2fa51d-439b-425a-af0e-05a48b372205-tmp-dir\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:37.536534 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.536520    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2de82e27-28f6-4ee2-bf84-74b92409a9b8-cert\") pod \"ingress-canary-r7pf8\" (UID: \"2de82e27-28f6-4ee2-bf84-74b92409a9b8\") " pod="openshift-ingress-canary/ingress-canary-r7pf8"
Sep 18 16:05:37.536752 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.536562    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svsgm\" (UniqueName: \"kubernetes.io/projected/ad2fa51d-439b-425a-af0e-05a48b372205-kube-api-access-svsgm\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:37.536752 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.536618    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slpvv\" (UniqueName: \"kubernetes.io/projected/2de82e27-28f6-4ee2-bf84-74b92409a9b8-kube-api-access-slpvv\") pod \"ingress-canary-r7pf8\" (UID: \"2de82e27-28f6-4ee2-bf84-74b92409a9b8\") " pod="openshift-ingress-canary/ingress-canary-r7pf8"
Sep 18 16:05:37.536752 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.536684    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2fa51d-439b-425a-af0e-05a48b372205-config-volume\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:37.596922 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.596891    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:37.596922 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.596888    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:37.597130 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.597057    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:37.599837 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.599812    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:05:37.600042 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.599834    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:05:37.600042 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.599896    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:05:37.600042 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.599907    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j9g75\""
Sep 18 16:05:37.600042 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.599842    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:05:37.600275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.600247    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89djg\""
Sep 18 16:05:37.637266 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.637248    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svsgm\" (UniqueName: \"kubernetes.io/projected/ad2fa51d-439b-425a-af0e-05a48b372205-kube-api-access-svsgm\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:37.637383 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.637292    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slpvv\" (UniqueName: \"kubernetes.io/projected/2de82e27-28f6-4ee2-bf84-74b92409a9b8-kube-api-access-slpvv\") pod \"ingress-canary-r7pf8\" (UID: \"2de82e27-28f6-4ee2-bf84-74b92409a9b8\") " pod="openshift-ingress-canary/ingress-canary-r7pf8"
Sep 18 16:05:37.637477 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.637456    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2fa51d-439b-425a-af0e-05a48b372205-config-volume\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:37.637556 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.637507    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:37.637612 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.637576    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad2fa51d-439b-425a-af0e-05a48b372205-tmp-dir\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:37.637675 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.637621    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2de82e27-28f6-4ee2-bf84-74b92409a9b8-cert\") pod \"ingress-canary-r7pf8\" (UID: \"2de82e27-28f6-4ee2-bf84-74b92409a9b8\") " pod="openshift-ingress-canary/ingress-canary-r7pf8"
Sep 18 16:05:37.637675 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:37.637593    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:05:37.637759 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:37.637716    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls podName:ad2fa51d-439b-425a-af0e-05a48b372205 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:38.137689679 +0000 UTC m=+34.046470021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls") pod "dns-default-c7lb2" (UID: "ad2fa51d-439b-425a-af0e-05a48b372205") : secret "dns-default-metrics-tls" not found
Sep 18 16:05:37.637990 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.637967    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad2fa51d-439b-425a-af0e-05a48b372205-tmp-dir\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:37.638158 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.638142    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2fa51d-439b-425a-af0e-05a48b372205-config-volume\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:37.642572 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.642554    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2de82e27-28f6-4ee2-bf84-74b92409a9b8-cert\") pod \"ingress-canary-r7pf8\" (UID: \"2de82e27-28f6-4ee2-bf84-74b92409a9b8\") " pod="openshift-ingress-canary/ingress-canary-r7pf8"
Sep 18 16:05:37.646375 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.646309    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svsgm\" (UniqueName: \"kubernetes.io/projected/ad2fa51d-439b-425a-af0e-05a48b372205-kube-api-access-svsgm\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:37.646603 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.646580    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slpvv\" (UniqueName: \"kubernetes.io/projected/2de82e27-28f6-4ee2-bf84-74b92409a9b8-kube-api-access-slpvv\") pod \"ingress-canary-r7pf8\" (UID: \"2de82e27-28f6-4ee2-bf84-74b92409a9b8\") " pod="openshift-ingress-canary/ingress-canary-r7pf8"
Sep 18 16:05:37.740059 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:37.740035    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r7pf8"
Sep 18 16:05:38.140920 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:38.140877    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:38.141109 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:38.141058    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:05:38.141175 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:38.141123    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls podName:ad2fa51d-439b-425a-af0e-05a48b372205 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:39.141104803 +0000 UTC m=+35.049885132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls") pod "dns-default-c7lb2" (UID: "ad2fa51d-439b-425a-af0e-05a48b372205") : secret "dns-default-metrics-tls" not found
Sep 18 16:05:38.241996 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:38.241956    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:05:38.242190 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:38.242118    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:05:38.242190 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:38.242182    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs podName:1f737909-4b95-481b-b092-0816b5be6f34 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:10.242167609 +0000 UTC m=+66.150947938 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs") pod "network-metrics-daemon-7h8rh" (UID: "1f737909-4b95-481b-b092-0816b5be6f34") : secret "metrics-daemon-secret" not found
Sep 18 16:05:38.343285 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:38.343251    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:38.345946 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:38.345914    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7cfa77c5-aa51-41ef-9673-c23c39835582-original-pull-secret\") pod \"global-pull-secret-syncer-pcmn7\" (UID: \"7cfa77c5-aa51-41ef-9673-c23c39835582\") " pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:38.444384 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:38.444346    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dps4t\" (UniqueName: \"kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t\") pod \"network-check-target-p69sq\" (UID: \"e8acfc8b-7c80-4522-9d84-b26a033bc57e\") " pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:38.446952 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:38.446925    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dps4t\" (UniqueName: \"kubernetes.io/projected/e8acfc8b-7c80-4522-9d84-b26a033bc57e-kube-api-access-dps4t\") pod \"network-check-target-p69sq\" (UID: \"e8acfc8b-7c80-4522-9d84-b26a033bc57e\") " pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:38.514036 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:38.513985    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:38.519120 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:38.519097    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pcmn7"
Sep 18 16:05:39.149760 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:39.149600    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:39.149760 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:39.149747    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:05:39.149880 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:39.149816    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls podName:ad2fa51d-439b-425a-af0e-05a48b372205 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:41.149799839 +0000 UTC m=+37.058580168 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls") pod "dns-default-c7lb2" (UID: "ad2fa51d-439b-425a-af0e-05a48b372205") : secret "dns-default-metrics-tls" not found
Sep 18 16:05:39.182343 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:39.182264    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r7pf8"]
Sep 18 16:05:39.185220 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:39.185193    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p69sq"]
Sep 18 16:05:39.185907 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:39.185875    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pcmn7"]
Sep 18 16:05:39.188834 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:39.188813    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8acfc8b_7c80_4522_9d84_b26a033bc57e.slice/crio-3d64a33727938d96b2f28b8ad87a774efe4fd3e335003da646ceb7e694af446e WatchSource:0}: Error finding container 3d64a33727938d96b2f28b8ad87a774efe4fd3e335003da646ceb7e694af446e: Status 404 returned error can't find the container with id 3d64a33727938d96b2f28b8ad87a774efe4fd3e335003da646ceb7e694af446e
Sep 18 16:05:39.189438 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:39.189324    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfa77c5_aa51_41ef_9673_c23c39835582.slice/crio-fb8dd824452282b5bb3d026b8dd7edc757be4346e987269da90276938b2615f4 WatchSource:0}: Error finding container fb8dd824452282b5bb3d026b8dd7edc757be4346e987269da90276938b2615f4: Status 404 returned error can't find the container with id fb8dd824452282b5bb3d026b8dd7edc757be4346e987269da90276938b2615f4
Sep 18 16:05:39.190230 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:05:39.190198    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de82e27_28f6_4ee2_bf84_74b92409a9b8.slice/crio-f24e6088b69a0b124871559b561114fbdcfa30605c4917df62390a62fb522fb3 WatchSource:0}: Error finding container f24e6088b69a0b124871559b561114fbdcfa30605c4917df62390a62fb522fb3: Status 404 returned error can't find the container with id f24e6088b69a0b124871559b561114fbdcfa30605c4917df62390a62fb522fb3
Sep 18 16:05:39.784214 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:39.784184    2546 generic.go:356] "Generic (PLEG): container finished" podID="7ea7d464-a8e8-4c9e-b437-93784c147707" containerID="fb7fb59f985b5afa0e06b3e2d2d807cfd6af8937232fc20d813858b4a53c7101" exitCode=0
Sep 18 16:05:39.784895 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:39.784249    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnmct" event={"ID":"7ea7d464-a8e8-4c9e-b437-93784c147707","Type":"ContainerDied","Data":"fb7fb59f985b5afa0e06b3e2d2d807cfd6af8937232fc20d813858b4a53c7101"}
Sep 18 16:05:39.785546 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:39.785397    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p69sq" event={"ID":"e8acfc8b-7c80-4522-9d84-b26a033bc57e","Type":"ContainerStarted","Data":"3d64a33727938d96b2f28b8ad87a774efe4fd3e335003da646ceb7e694af446e"}
Sep 18 16:05:39.786467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:39.786442    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pcmn7" event={"ID":"7cfa77c5-aa51-41ef-9673-c23c39835582","Type":"ContainerStarted","Data":"fb8dd824452282b5bb3d026b8dd7edc757be4346e987269da90276938b2615f4"}
Sep 18 16:05:39.787568 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:39.787550    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r7pf8" event={"ID":"2de82e27-28f6-4ee2-bf84-74b92409a9b8","Type":"ContainerStarted","Data":"f24e6088b69a0b124871559b561114fbdcfa30605c4917df62390a62fb522fb3"}
Sep 18 16:05:40.796859 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:40.795974    2546 generic.go:356] "Generic (PLEG): container finished" podID="7ea7d464-a8e8-4c9e-b437-93784c147707" containerID="8a4dafc4c421b4302025f83671e2441f9dac7e7b4877e5704fa945f4095ea97e" exitCode=0
Sep 18 16:05:40.796859 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:40.796045    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnmct" event={"ID":"7ea7d464-a8e8-4c9e-b437-93784c147707","Type":"ContainerDied","Data":"8a4dafc4c421b4302025f83671e2441f9dac7e7b4877e5704fa945f4095ea97e"}
Sep 18 16:05:41.169224 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:41.169097    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:41.169447 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:41.169277    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:05:41.169447 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:41.169364    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls podName:ad2fa51d-439b-425a-af0e-05a48b372205 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:45.16934182 +0000 UTC m=+41.078122164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls") pod "dns-default-c7lb2" (UID: "ad2fa51d-439b-425a-af0e-05a48b372205") : secret "dns-default-metrics-tls" not found
Sep 18 16:05:41.803757 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:41.803665    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qnmct" event={"ID":"7ea7d464-a8e8-4c9e-b437-93784c147707","Type":"ContainerStarted","Data":"b5d3f9e2ce95e4db2ee55d0f3e7d49d6c2ee4861abb9d73d93c9c7bfc556eb8e"}
Sep 18 16:05:41.828503 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:41.828407    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qnmct" podStartSLOduration=9.964775165 podStartE2EDuration="37.828387216s" totalImagesPullingTime="27.863612051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:41.826646765 +0000 UTC m=+37.735427140" watchObservedRunningTime="2026-09-18 16:05:41.828387216 +0000 UTC m=+37.737167568"
Sep 18 16:05:45.199225 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:45.199188    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:45.199532 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:45.199304    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:05:45.199532 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:45.199353    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls podName:ad2fa51d-439b-425a-af0e-05a48b372205 nodeName:}" failed. No retries permitted until 2026-09-18 16:05:53.199340709 +0000 UTC m=+49.108121039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls") pod "dns-default-c7lb2" (UID: "ad2fa51d-439b-425a-af0e-05a48b372205") : secret "dns-default-metrics-tls" not found
Sep 18 16:05:45.815215 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:45.815178    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p69sq" event={"ID":"e8acfc8b-7c80-4522-9d84-b26a033bc57e","Type":"ContainerStarted","Data":"8421630e0ecf91b8e2175641be55a468078665b9f3cc96d990ab06c6f5021270"}
Sep 18 16:05:45.815394 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:45.815275    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:05:45.816665 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:45.816641    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pcmn7" event={"ID":"7cfa77c5-aa51-41ef-9673-c23c39835582","Type":"ContainerStarted","Data":"2f21393b50759c588784373376c9ef23858b37dcac6361ac1062fc36ab4ee052"}
Sep 18 16:05:45.818066 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:45.818047    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r7pf8" event={"ID":"2de82e27-28f6-4ee2-bf84-74b92409a9b8","Type":"ContainerStarted","Data":"080118cf046e165e3c97161d2ec4b4ff632023e032886b45b646d13c70af1742"}
Sep 18 16:05:45.830854 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:45.830813    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p69sq" podStartSLOduration=35.725640027 podStartE2EDuration="41.830801084s" totalImagesPullingTime="6.105161057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:45.829935329 +0000 UTC m=+41.738715699" watchObservedRunningTime="2026-09-18 16:05:45.830801084 +0000 UTC m=+41.739581450"
Sep 18 16:05:45.844801 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:45.844765    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pcmn7" podStartSLOduration=35.729720621 podStartE2EDuration="41.844729824s" totalImagesPullingTime="6.115009203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:45.843749832 +0000 UTC m=+41.752530208" watchObservedRunningTime="2026-09-18 16:05:45.844729824 +0000 UTC m=+41.753510228"
Sep 18 16:05:53.249533 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:05:53.249483    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:05:53.249970 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:53.249630    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:05:53.249970 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:05:53.249687    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls podName:ad2fa51d-439b-425a-af0e-05a48b372205 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:09.249672769 +0000 UTC m=+65.158453104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls") pod "dns-default-c7lb2" (UID: "ad2fa51d-439b-425a-af0e-05a48b372205") : secret "dns-default-metrics-tls" not found
Sep 18 16:06:02.779314 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:02.779285    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnn6j"
Sep 18 16:06:02.805938 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:02.805887    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r7pf8" podStartSLOduration=19.702110244 podStartE2EDuration="25.805874039s" totalImagesPullingTime="6.103763795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:05:45.85747166 +0000 UTC m=+41.766252011" watchObservedRunningTime="2026-09-18 16:06:02.805874039 +0000 UTC m=+58.714654390"
Sep 18 16:06:09.255280 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:09.255243    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:06:09.255740 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:09.255367    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:06:09.255740 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:09.255435    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls podName:ad2fa51d-439b-425a-af0e-05a48b372205 nodeName:}" failed. No retries permitted until 2026-09-18 16:06:41.255421931 +0000 UTC m=+97.164202261 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls") pod "dns-default-c7lb2" (UID: "ad2fa51d-439b-425a-af0e-05a48b372205") : secret "dns-default-metrics-tls" not found
Sep 18 16:06:10.262987 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:10.262948    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:06:10.263355 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:10.263110    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:06:10.263355 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:10.263184    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs podName:1f737909-4b95-481b-b092-0816b5be6f34 nodeName:}" failed. No retries permitted until 2026-09-18 16:07:14.263168575 +0000 UTC m=+130.171948906 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs") pod "network-metrics-daemon-7h8rh" (UID: "1f737909-4b95-481b-b092-0816b5be6f34") : secret "metrics-daemon-secret" not found
Sep 18 16:06:16.823376 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:16.823344    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p69sq"
Sep 18 16:06:41.270036 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:41.269977    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:06:41.270389 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:41.270118    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:06:41.270389 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:41.270190    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls podName:ad2fa51d-439b-425a-af0e-05a48b372205 nodeName:}" failed. No retries permitted until 2026-09-18 16:07:45.270173465 +0000 UTC m=+161.178953795 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls") pod "dns-default-c7lb2" (UID: "ad2fa51d-439b-425a-af0e-05a48b372205") : secret "dns-default-metrics-tls" not found
Sep 18 16:06:45.546753 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.546714    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-9cjph"]
Sep 18 16:06:45.548961 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.548943    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9cjph"
Sep 18 16:06:45.551302 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.551282    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zcx4s\""
Sep 18 16:06:45.558856 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.558829    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-9cjph"]
Sep 18 16:06:45.565236 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.565206    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-847f74c87d-nnkct"]
Sep 18 16:06:45.566816 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.566800    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.569298 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.569274    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 16:06:45.569423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.569310    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 16:06:45.570002 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.569974    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 16:06:45.570002 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.569983    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 16:06:45.570160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.570018    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 16:06:45.570160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.569975    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 16:06:45.570790 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.570773    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wckmk\""
Sep 18 16:06:45.582642 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.582608    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-847f74c87d-nnkct"]
Sep 18 16:06:45.597499 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.597466    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k7s4\" (UniqueName: \"kubernetes.io/projected/4ad78489-ae4c-47db-adc3-d529059de873-kube-api-access-2k7s4\") pod \"network-check-source-7cbc45fddc-9cjph\" (UID: \"4ad78489-ae4c-47db-adc3-d529059de873\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9cjph"
Sep 18 16:06:45.657486 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.657444    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"]
Sep 18 16:06:45.659434 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.659410    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"
Sep 18 16:06:45.660791 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.660768    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"]
Sep 18 16:06:45.662323 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.662309    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:06:45.671771 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.671692    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 16:06:45.672505 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.672490    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 16:06:45.676224 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.676206    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 16:06:45.676341 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.676211    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 16:06:45.690274 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.690247    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:06:45.690422 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.690374    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 16:06:45.690879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.690866    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-p9mkr\""
Sep 18 16:06:45.690921 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.690897    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 16:06:45.692038 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.692001    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"]
Sep 18 16:06:45.692862 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.692843    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 16:06:45.693142 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.693126    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-p7v7w\""
Sep 18 16:06:45.698431 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.698405    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k7s4\" (UniqueName: \"kubernetes.io/projected/4ad78489-ae4c-47db-adc3-d529059de873-kube-api-access-2k7s4\") pod \"network-check-source-7cbc45fddc-9cjph\" (UID: \"4ad78489-ae4c-47db-adc3-d529059de873\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9cjph"
Sep 18 16:06:45.698564 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.698528    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.698732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.698712    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-default-certificate\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.698801 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.698741    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.698801 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.698759    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-stats-auth\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.699061 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.698801    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjk7p\" (UniqueName: \"kubernetes.io/projected/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-kube-api-access-xjk7p\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.699662 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.699642    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"]
Sep 18 16:06:45.724084 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.724056    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k7s4\" (UniqueName: \"kubernetes.io/projected/4ad78489-ae4c-47db-adc3-d529059de873-kube-api-access-2k7s4\") pod \"network-check-source-7cbc45fddc-9cjph\" (UID: \"4ad78489-ae4c-47db-adc3-d529059de873\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9cjph"
Sep 18 16:06:45.799739 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.799655    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.799739 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.799692    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9ec0ffc7-1d21-4f0d-be23-507156419b8b-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:06:45.799739 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.799719    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e62eede-ea1b-415e-8549-3df93215459e-serving-cert\") pod \"service-ca-operator-57cdc4b768-7sgr7\" (UID: \"8e62eede-ea1b-415e-8549-3df93215459e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"
Sep 18 16:06:45.799997 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.799779    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-default-certificate\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.799997 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.799805    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.799997 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.799827    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-stats-auth\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.799997 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.799850    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tplp\" (UniqueName: \"kubernetes.io/projected/8e62eede-ea1b-415e-8549-3df93215459e-kube-api-access-5tplp\") pod \"service-ca-operator-57cdc4b768-7sgr7\" (UID: \"8e62eede-ea1b-415e-8549-3df93215459e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"
Sep 18 16:06:45.799997 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.799869    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjk7p\" (UniqueName: \"kubernetes.io/projected/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-kube-api-access-xjk7p\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.799997 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:45.799824    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:06:45.799997 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.799892    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e62eede-ea1b-415e-8549-3df93215459e-config\") pod \"service-ca-operator-57cdc4b768-7sgr7\" (UID: \"8e62eede-ea1b-415e-8549-3df93215459e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"
Sep 18 16:06:45.799997 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.799915    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:06:45.799997 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:45.799935    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle podName:fdf89ce0-2abf-4a51-a332-8c9c01121c6a nodeName:}" failed. No retries permitted until 2026-09-18 16:06:46.299916368 +0000 UTC m=+102.208696699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle") pod "router-default-847f74c87d-nnkct" (UID: "fdf89ce0-2abf-4a51-a332-8c9c01121c6a") : configmap references non-existent config key: service-ca.crt
Sep 18 16:06:45.799997 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:45.799966    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs podName:fdf89ce0-2abf-4a51-a332-8c9c01121c6a nodeName:}" failed. No retries permitted until 2026-09-18 16:06:46.299955384 +0000 UTC m=+102.208735715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs") pod "router-default-847f74c87d-nnkct" (UID: "fdf89ce0-2abf-4a51-a332-8c9c01121c6a") : secret "router-metrics-certs-default" not found
Sep 18 16:06:45.800385 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.799994    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v94f\" (UniqueName: \"kubernetes.io/projected/9ec0ffc7-1d21-4f0d-be23-507156419b8b-kube-api-access-4v94f\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:06:45.802260 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.802242    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-default-certificate\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.802463 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.802443    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-stats-auth\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.808541 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.808519    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjk7p\" (UniqueName: \"kubernetes.io/projected/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-kube-api-access-xjk7p\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:45.858046 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.857989    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9cjph"
Sep 18 16:06:45.901304 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.901270    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tplp\" (UniqueName: \"kubernetes.io/projected/8e62eede-ea1b-415e-8549-3df93215459e-kube-api-access-5tplp\") pod \"service-ca-operator-57cdc4b768-7sgr7\" (UID: \"8e62eede-ea1b-415e-8549-3df93215459e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"
Sep 18 16:06:45.901304 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.901309    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e62eede-ea1b-415e-8549-3df93215459e-config\") pod \"service-ca-operator-57cdc4b768-7sgr7\" (UID: \"8e62eede-ea1b-415e-8549-3df93215459e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"
Sep 18 16:06:45.901564 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.901336    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:06:45.901564 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.901363    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v94f\" (UniqueName: \"kubernetes.io/projected/9ec0ffc7-1d21-4f0d-be23-507156419b8b-kube-api-access-4v94f\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:06:45.901564 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.901430    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9ec0ffc7-1d21-4f0d-be23-507156419b8b-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:06:45.901564 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.901465    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e62eede-ea1b-415e-8549-3df93215459e-serving-cert\") pod \"service-ca-operator-57cdc4b768-7sgr7\" (UID: \"8e62eede-ea1b-415e-8549-3df93215459e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"
Sep 18 16:06:45.901564 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:45.901490    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:06:45.901564 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:45.901554    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls podName:9ec0ffc7-1d21-4f0d-be23-507156419b8b nodeName:}" failed. No retries permitted until 2026-09-18 16:06:46.401535759 +0000 UTC m=+102.310316110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-vwnjj" (UID: "9ec0ffc7-1d21-4f0d-be23-507156419b8b") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:06:45.901979 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.901950    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8e62eede-ea1b-415e-8549-3df93215459e-config\") pod \"service-ca-operator-57cdc4b768-7sgr7\" (UID: \"8e62eede-ea1b-415e-8549-3df93215459e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"
Sep 18 16:06:45.902365 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.902330    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9ec0ffc7-1d21-4f0d-be23-507156419b8b-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:06:45.903966 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.903930    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e62eede-ea1b-415e-8549-3df93215459e-serving-cert\") pod \"service-ca-operator-57cdc4b768-7sgr7\" (UID: \"8e62eede-ea1b-415e-8549-3df93215459e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"
Sep 18 16:06:45.911309 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.911281    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tplp\" (UniqueName: \"kubernetes.io/projected/8e62eede-ea1b-415e-8549-3df93215459e-kube-api-access-5tplp\") pod \"service-ca-operator-57cdc4b768-7sgr7\" (UID: \"8e62eede-ea1b-415e-8549-3df93215459e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"
Sep 18 16:06:45.911445 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.911328    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v94f\" (UniqueName: \"kubernetes.io/projected/9ec0ffc7-1d21-4f0d-be23-507156419b8b-kube-api-access-4v94f\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:06:45.970092 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:45.970063    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"
Sep 18 16:06:46.127078 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:46.127053    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-9cjph"]
Sep 18 16:06:46.129664 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:06:46.129633    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad78489_ae4c_47db_adc3_d529059de873.slice/crio-11a5f6d33c34f204d93f5acb0f57fadbeb3bcbe6e78b2da9dbd0568ca6793097 WatchSource:0}: Error finding container 11a5f6d33c34f204d93f5acb0f57fadbeb3bcbe6e78b2da9dbd0568ca6793097: Status 404 returned error can't find the container with id 11a5f6d33c34f204d93f5acb0f57fadbeb3bcbe6e78b2da9dbd0568ca6793097
Sep 18 16:06:46.237771 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:46.237740    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7"]
Sep 18 16:06:46.240567 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:06:46.240538    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e62eede_ea1b_415e_8549_3df93215459e.slice/crio-0c0e442ff9c97320be029f4a7de0154dd725998769eba632836fe03925a8fee0 WatchSource:0}: Error finding container 0c0e442ff9c97320be029f4a7de0154dd725998769eba632836fe03925a8fee0: Status 404 returned error can't find the container with id 0c0e442ff9c97320be029f4a7de0154dd725998769eba632836fe03925a8fee0
Sep 18 16:06:46.304279 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:46.304238    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:46.304447 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:46.304328    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:46.304447 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:46.304432    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle podName:fdf89ce0-2abf-4a51-a332-8c9c01121c6a nodeName:}" failed. No retries permitted until 2026-09-18 16:06:47.304408726 +0000 UTC m=+103.213189060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle") pod "router-default-847f74c87d-nnkct" (UID: "fdf89ce0-2abf-4a51-a332-8c9c01121c6a") : configmap references non-existent config key: service-ca.crt
Sep 18 16:06:46.304572 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:46.304440    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:06:46.304572 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:46.304486    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs podName:fdf89ce0-2abf-4a51-a332-8c9c01121c6a nodeName:}" failed. No retries permitted until 2026-09-18 16:06:47.304475952 +0000 UTC m=+103.213256284 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs") pod "router-default-847f74c87d-nnkct" (UID: "fdf89ce0-2abf-4a51-a332-8c9c01121c6a") : secret "router-metrics-certs-default" not found
Sep 18 16:06:46.405302 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:46.405212    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:06:46.405437 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:46.405336    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:06:46.405437 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:46.405392    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls podName:9ec0ffc7-1d21-4f0d-be23-507156419b8b nodeName:}" failed. No retries permitted until 2026-09-18 16:06:47.405376542 +0000 UTC m=+103.314156871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-vwnjj" (UID: "9ec0ffc7-1d21-4f0d-be23-507156419b8b") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:06:46.970779 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:46.970700    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7" event={"ID":"8e62eede-ea1b-415e-8549-3df93215459e","Type":"ContainerStarted","Data":"0c0e442ff9c97320be029f4a7de0154dd725998769eba632836fe03925a8fee0"}
Sep 18 16:06:46.972590 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:46.972557    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9cjph" event={"ID":"4ad78489-ae4c-47db-adc3-d529059de873","Type":"ContainerStarted","Data":"25db85aadbb77e7f2687ecb4d8beeba437787fb796a1d3bfaa08ad26a7058732"}
Sep 18 16:06:46.972722 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:46.972608    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9cjph" event={"ID":"4ad78489-ae4c-47db-adc3-d529059de873","Type":"ContainerStarted","Data":"11a5f6d33c34f204d93f5acb0f57fadbeb3bcbe6e78b2da9dbd0568ca6793097"}
Sep 18 16:06:46.990651 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:46.990594    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-9cjph" podStartSLOduration=1.990572939 podStartE2EDuration="1.990572939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:06:46.989207754 +0000 UTC m=+102.897988112" watchObservedRunningTime="2026-09-18 16:06:46.990572939 +0000 UTC m=+102.899353293"
Sep 18 16:06:47.311796 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:47.311700    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:47.311980 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:47.311806    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:47.311980 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:47.311926    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:06:47.311980 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:47.311956    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle podName:fdf89ce0-2abf-4a51-a332-8c9c01121c6a nodeName:}" failed. No retries permitted until 2026-09-18 16:06:49.311931752 +0000 UTC m=+105.220712097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle") pod "router-default-847f74c87d-nnkct" (UID: "fdf89ce0-2abf-4a51-a332-8c9c01121c6a") : configmap references non-existent config key: service-ca.crt
Sep 18 16:06:47.312172 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:47.311994    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs podName:fdf89ce0-2abf-4a51-a332-8c9c01121c6a nodeName:}" failed. No retries permitted until 2026-09-18 16:06:49.311983236 +0000 UTC m=+105.220763571 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs") pod "router-default-847f74c87d-nnkct" (UID: "fdf89ce0-2abf-4a51-a332-8c9c01121c6a") : secret "router-metrics-certs-default" not found
Sep 18 16:06:47.412788 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:47.412752    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:06:47.412988 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:47.412913    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:06:47.413070 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:47.412992    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls podName:9ec0ffc7-1d21-4f0d-be23-507156419b8b nodeName:}" failed. No retries permitted until 2026-09-18 16:06:49.412974533 +0000 UTC m=+105.321754863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-vwnjj" (UID: "9ec0ffc7-1d21-4f0d-be23-507156419b8b") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:06:48.979748 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:48.979713    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7" event={"ID":"8e62eede-ea1b-415e-8549-3df93215459e","Type":"ContainerStarted","Data":"3cb91fd86824f74c5df2771e2af0257629209ab209887c473a880e2367c2d2c5"}
Sep 18 16:06:48.998869 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:48.998817    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7" podStartSLOduration=1.840699079 podStartE2EDuration="3.998800162s" totalImagesPullingTime="2.158101083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:06:48.997955731 +0000 UTC m=+104.906736112" watchObservedRunningTime="2026-09-18 16:06:48.998800162 +0000 UTC m=+104.907580513"
Sep 18 16:06:49.328367 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:49.328250    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:49.328367 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:49.328348    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:49.328593 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:49.328448    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle podName:fdf89ce0-2abf-4a51-a332-8c9c01121c6a nodeName:}" failed. No retries permitted until 2026-09-18 16:06:53.328425739 +0000 UTC m=+109.237206069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle") pod "router-default-847f74c87d-nnkct" (UID: "fdf89ce0-2abf-4a51-a332-8c9c01121c6a") : configmap references non-existent config key: service-ca.crt
Sep 18 16:06:49.328593 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:49.328450    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:06:49.328593 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:49.328499    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs podName:fdf89ce0-2abf-4a51-a332-8c9c01121c6a nodeName:}" failed. No retries permitted until 2026-09-18 16:06:53.328490046 +0000 UTC m=+109.237270391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs") pod "router-default-847f74c87d-nnkct" (UID: "fdf89ce0-2abf-4a51-a332-8c9c01121c6a") : secret "router-metrics-certs-default" not found
Sep 18 16:06:49.429439 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:49.429385    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:06:49.429599 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:49.429573    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:06:49.429671 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:49.429661    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls podName:9ec0ffc7-1d21-4f0d-be23-507156419b8b nodeName:}" failed. No retries permitted until 2026-09-18 16:06:53.429640326 +0000 UTC m=+109.338420659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-vwnjj" (UID: "9ec0ffc7-1d21-4f0d-be23-507156419b8b") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:06:51.495873 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.495840    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-69shv"]
Sep 18 16:06:51.497845 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.497816    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:51.500665 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.500647    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 16:06:51.500774 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.500716    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 16:06:51.501577 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.501557    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 16:06:51.501577 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.501570    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 16:06:51.501712 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.501580    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnksp\""
Sep 18 16:06:51.512567 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.512543    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-69shv"]
Sep 18 16:06:51.644590 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.644546    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-data-volume\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:51.644590 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.644592    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:51.644840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.644676    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:51.644840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.644695    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-crio-socket\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:51.644840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.644710    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcd5v\" (UniqueName: \"kubernetes.io/projected/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-api-access-lcd5v\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:51.746118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.746004    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-data-volume\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:51.746118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.746067    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:51.746337 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:51.746166    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:51.746337 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.746181    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:51.746337 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.746207    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-crio-socket\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:51.746337 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:51.746230    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls podName:a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b nodeName:}" failed. No retries permitted until 2026-09-18 16:06:52.246215173 +0000 UTC m=+108.154995534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69shv" (UID: "a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:51.746337 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.746254    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcd5v\" (UniqueName: \"kubernetes.io/projected/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-api-access-lcd5v\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:51.746337 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.746263    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-crio-socket\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:51.746337 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:51.746315    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:06:51.746337 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:51.746341    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm podName:a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b nodeName:}" failed. No retries permitted until 2026-09-18 16:06:52.246332408 +0000 UTC m=+108.155112738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69shv" (UID: "a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b") : configmap "kube-rbac-proxy" not found
Sep 18 16:06:51.746600 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.746380    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-data-volume\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:51.761130 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:51.761102    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcd5v\" (UniqueName: \"kubernetes.io/projected/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-api-access-lcd5v\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:52.047477 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.047398    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-7xldd"]
Sep 18 16:06:52.049470 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.049451    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd"
Sep 18 16:06:52.052510 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.052490    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 18 16:06:52.053460 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.053430    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 18 16:06:52.053549 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.053430    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 16:06:52.053549 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.053493    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 18 16:06:52.053549 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.053544    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nvjfq\""
Sep 18 16:06:52.065160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.065136    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-7xldd"]
Sep 18 16:06:52.149348 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.149307    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfbg7\" (UniqueName: \"kubernetes.io/projected/45b59ace-62a2-45ae-9f58-a3d21c8c2ab5-kube-api-access-pfbg7\") pod \"service-ca-55f6fc7b86-7xldd\" (UID: \"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd"
Sep 18 16:06:52.149544 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.149360    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/45b59ace-62a2-45ae-9f58-a3d21c8c2ab5-signing-cabundle\") pod \"service-ca-55f6fc7b86-7xldd\" (UID: \"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd"
Sep 18 16:06:52.149544 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.149427    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/45b59ace-62a2-45ae-9f58-a3d21c8c2ab5-signing-key\") pod \"service-ca-55f6fc7b86-7xldd\" (UID: \"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd"
Sep 18 16:06:52.250805 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.250763    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/45b59ace-62a2-45ae-9f58-a3d21c8c2ab5-signing-cabundle\") pod \"service-ca-55f6fc7b86-7xldd\" (UID: \"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd"
Sep 18 16:06:52.250805 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.250805    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:52.251011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.250824    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/45b59ace-62a2-45ae-9f58-a3d21c8c2ab5-signing-key\") pod \"service-ca-55f6fc7b86-7xldd\" (UID: \"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd"
Sep 18 16:06:52.251011 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:52.250935    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:52.251126 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:52.251011    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls podName:a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b nodeName:}" failed. No retries permitted until 2026-09-18 16:06:53.250993801 +0000 UTC m=+109.159774150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69shv" (UID: "a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:52.251126 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.251087    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:52.251126 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.251119    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfbg7\" (UniqueName: \"kubernetes.io/projected/45b59ace-62a2-45ae-9f58-a3d21c8c2ab5-kube-api-access-pfbg7\") pod \"service-ca-55f6fc7b86-7xldd\" (UID: \"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd"
Sep 18 16:06:52.251273 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:52.251184    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:06:52.251273 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:52.251256    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm podName:a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b nodeName:}" failed. No retries permitted until 2026-09-18 16:06:53.251240641 +0000 UTC m=+109.160020991 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69shv" (UID: "a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b") : configmap "kube-rbac-proxy" not found
Sep 18 16:06:52.251442 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.251421    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/45b59ace-62a2-45ae-9f58-a3d21c8c2ab5-signing-cabundle\") pod \"service-ca-55f6fc7b86-7xldd\" (UID: \"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd"
Sep 18 16:06:52.253261 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.253232    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/45b59ace-62a2-45ae-9f58-a3d21c8c2ab5-signing-key\") pod \"service-ca-55f6fc7b86-7xldd\" (UID: \"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd"
Sep 18 16:06:52.259863 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.259838    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfbg7\" (UniqueName: \"kubernetes.io/projected/45b59ace-62a2-45ae-9f58-a3d21c8c2ab5-kube-api-access-pfbg7\") pod \"service-ca-55f6fc7b86-7xldd\" (UID: \"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5\") " pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd"
Sep 18 16:06:52.359287 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.359202    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd"
Sep 18 16:06:52.627287 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.627185    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-7xldd"]
Sep 18 16:06:52.629906 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:06:52.629879    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b59ace_62a2_45ae_9f58_a3d21c8c2ab5.slice/crio-3cdb97ddfb2e41180da91a99dca920089083aff6edb6e1cc7968ebfd732afc22 WatchSource:0}: Error finding container 3cdb97ddfb2e41180da91a99dca920089083aff6edb6e1cc7968ebfd732afc22: Status 404 returned error can't find the container with id 3cdb97ddfb2e41180da91a99dca920089083aff6edb6e1cc7968ebfd732afc22
Sep 18 16:06:52.992613 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.992579    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd" event={"ID":"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5","Type":"ContainerStarted","Data":"054d86832b7dfca108df9e1685fffa11e2b3ce1838b0b67ce7c244be56696877"}
Sep 18 16:06:52.992778 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:52.992630    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd" event={"ID":"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5","Type":"ContainerStarted","Data":"3cdb97ddfb2e41180da91a99dca920089083aff6edb6e1cc7968ebfd732afc22"}
Sep 18 16:06:53.009744 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:53.009695    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd" podStartSLOduration=1.009679369 podStartE2EDuration="1.009679369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:06:53.008964439 +0000 UTC m=+108.917744815" watchObservedRunningTime="2026-09-18 16:06:53.009679369 +0000 UTC m=+108.918459751"
Sep 18 16:06:53.259506 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:53.259415    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:53.259654 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:53.259525    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:53.259654 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:53.259590    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:53.259654 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:53.259606    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:06:53.259750 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:53.259672    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls podName:a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b nodeName:}" failed. No retries permitted until 2026-09-18 16:06:55.259650768 +0000 UTC m=+111.168431104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69shv" (UID: "a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:53.259750 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:53.259691    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm podName:a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b nodeName:}" failed. No retries permitted until 2026-09-18 16:06:55.259681999 +0000 UTC m=+111.168462332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69shv" (UID: "a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b") : configmap "kube-rbac-proxy" not found
Sep 18 16:06:53.360291 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:53.360260    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:53.360443 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:53.360324    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:06:53.360443 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:53.360405    2546 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:06:53.360443 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:53.360438    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle podName:fdf89ce0-2abf-4a51-a332-8c9c01121c6a nodeName:}" failed. No retries permitted until 2026-09-18 16:07:01.360425747 +0000 UTC m=+117.269206081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle") pod "router-default-847f74c87d-nnkct" (UID: "fdf89ce0-2abf-4a51-a332-8c9c01121c6a") : configmap references non-existent config key: service-ca.crt
Sep 18 16:06:53.360546 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:53.360465    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs podName:fdf89ce0-2abf-4a51-a332-8c9c01121c6a nodeName:}" failed. No retries permitted until 2026-09-18 16:07:01.360452635 +0000 UTC m=+117.269232970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs") pod "router-default-847f74c87d-nnkct" (UID: "fdf89ce0-2abf-4a51-a332-8c9c01121c6a") : secret "router-metrics-certs-default" not found
Sep 18 16:06:53.461655 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:53.461621    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:06:53.461819 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:53.461787    2546 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 16:06:53.461890 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:53.461865    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls podName:9ec0ffc7-1d21-4f0d-be23-507156419b8b nodeName:}" failed. No retries permitted until 2026-09-18 16:07:01.461844434 +0000 UTC m=+117.370624781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-vwnjj" (UID: "9ec0ffc7-1d21-4f0d-be23-507156419b8b") : secret "cluster-monitoring-operator-tls" not found
Sep 18 16:06:54.445443 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:54.445410    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2bl6_2023cf66-25c9-441a-b1b9-6892cf0b2746/dns-node-resolver/0.log"
Sep 18 16:06:55.245412 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:55.245380    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6gkbl_49ffc9f3-7c5f-48bc-8217-838ac6651979/node-ca/0.log"
Sep 18 16:06:55.276943 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:55.276909    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:55.277049 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:55.276983    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:55.277133 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:55.277020    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:06:55.277133 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:55.277112    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm podName:a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b nodeName:}" failed. No retries permitted until 2026-09-18 16:06:59.2770946 +0000 UTC m=+115.185874936 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69shv" (UID: "a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b") : configmap "kube-rbac-proxy" not found
Sep 18 16:06:55.277248 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:55.277141    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:55.277248 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:55.277210    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls podName:a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b nodeName:}" failed. No retries permitted until 2026-09-18 16:06:59.277191855 +0000 UTC m=+115.185972186 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69shv" (UID: "a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:56.246091 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:56.246054    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r7pf8_2de82e27-28f6-4ee2-bf84-74b92409a9b8/serve-healthcheck-canary/0.log"
Sep 18 16:06:59.307128 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:59.307087    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:59.307590 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:06:59.307213    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:06:59.307590 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:59.307229    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:59.307590 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:59.307291    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls podName:a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b nodeName:}" failed. No retries permitted until 2026-09-18 16:07:07.307276633 +0000 UTC m=+123.216056963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69shv" (UID: "a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:06:59.307590 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:59.307308    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:06:59.307590 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:06:59.307359    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm podName:a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b nodeName:}" failed. No retries permitted until 2026-09-18 16:07:07.307343779 +0000 UTC m=+123.216124124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69shv" (UID: "a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b") : configmap "kube-rbac-proxy" not found
Sep 18 16:07:01.426700 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:01.426647    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:07:01.427121 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:01.426767    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:07:01.427533 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:01.427509    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-service-ca-bundle\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:07:01.429019 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:01.428997    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fdf89ce0-2abf-4a51-a332-8c9c01121c6a-metrics-certs\") pod \"router-default-847f74c87d-nnkct\" (UID: \"fdf89ce0-2abf-4a51-a332-8c9c01121c6a\") " pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:07:01.475373 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:01.475328    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:07:01.528237 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:01.528167    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:07:01.531751 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:01.531531    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9ec0ffc7-1d21-4f0d-be23-507156419b8b-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-vwnjj\" (UID: \"9ec0ffc7-1d21-4f0d-be23-507156419b8b\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:07:01.574117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:01.574082    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"
Sep 18 16:07:01.750160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:01.750136    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-847f74c87d-nnkct"]
Sep 18 16:07:01.752411 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:07:01.752375    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf89ce0_2abf_4a51_a332_8c9c01121c6a.slice/crio-78c1b43791fb30ba75e94e89404fb65f28633b5e3ed0f8382bcd88192b7488de WatchSource:0}: Error finding container 78c1b43791fb30ba75e94e89404fb65f28633b5e3ed0f8382bcd88192b7488de: Status 404 returned error can't find the container with id 78c1b43791fb30ba75e94e89404fb65f28633b5e3ed0f8382bcd88192b7488de
Sep 18 16:07:01.847161 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:01.847083    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj"]
Sep 18 16:07:01.850814 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:07:01.850785    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec0ffc7_1d21_4f0d_be23_507156419b8b.slice/crio-f8cd1d28c1896b69df8b6bbaf51825705e380a1bffca7ae1a5c0e93dc5e9cdc9 WatchSource:0}: Error finding container f8cd1d28c1896b69df8b6bbaf51825705e380a1bffca7ae1a5c0e93dc5e9cdc9: Status 404 returned error can't find the container with id f8cd1d28c1896b69df8b6bbaf51825705e380a1bffca7ae1a5c0e93dc5e9cdc9
Sep 18 16:07:02.023063 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:02.022945    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-847f74c87d-nnkct" event={"ID":"fdf89ce0-2abf-4a51-a332-8c9c01121c6a","Type":"ContainerStarted","Data":"9748c1e82233f28975a7246f1206ceacfafb15da93b6ef6c5f81f039b965cff6"}
Sep 18 16:07:02.023063 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:02.023015    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-847f74c87d-nnkct" event={"ID":"fdf89ce0-2abf-4a51-a332-8c9c01121c6a","Type":"ContainerStarted","Data":"78c1b43791fb30ba75e94e89404fb65f28633b5e3ed0f8382bcd88192b7488de"}
Sep 18 16:07:02.024070 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:02.024041    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj" event={"ID":"9ec0ffc7-1d21-4f0d-be23-507156419b8b","Type":"ContainerStarted","Data":"f8cd1d28c1896b69df8b6bbaf51825705e380a1bffca7ae1a5c0e93dc5e9cdc9"}
Sep 18 16:07:02.044040 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:02.043976    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-847f74c87d-nnkct" podStartSLOduration=17.043958903 podStartE2EDuration="17.043958903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:02.043147747 +0000 UTC m=+117.951928119" watchObservedRunningTime="2026-09-18 16:07:02.043958903 +0000 UTC m=+117.952739255"
Sep 18 16:07:02.476212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:02.476172    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:07:02.478773 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:02.478747    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:07:03.027699 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:03.027669    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:07:03.029160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:03.029137    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-847f74c87d-nnkct"
Sep 18 16:07:04.033135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:04.033047    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj" event={"ID":"9ec0ffc7-1d21-4f0d-be23-507156419b8b","Type":"ContainerStarted","Data":"9edee8a7f5a4f43f20608e2612b0eeefbd1ffc048aae76bb6c08a1d6aaece2ee"}
Sep 18 16:07:04.049573 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:04.049524    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-vwnjj" podStartSLOduration=17.125223951 podStartE2EDuration="19.04950905s" totalImagesPullingTime="1.924285099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:04.048699826 +0000 UTC m=+119.957480203" watchObservedRunningTime="2026-09-18 16:07:04.04950905 +0000 UTC m=+119.958289402"
Sep 18 16:07:07.376939 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:07.376883    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:07:07.377377 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:07.376957    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:07:07.377445 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:07.377421    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:07:07.379243 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:07.379219    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69shv\" (UID: \"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b\") " pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:07:07.409743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:07.409720    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnksp\""
Sep 18 16:07:07.417967 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:07.417949    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-69shv"
Sep 18 16:07:07.687722 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:07.687651    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-69shv"]
Sep 18 16:07:07.690812 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:07:07.690789    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83c2bf2_11a5_4eb2_b38e_d7e94cc4fe4b.slice/crio-e26ec9a4acd6ab6f67845de49db1972d8cbe1f4976df2fa3aeb23e6fe816bd68 WatchSource:0}: Error finding container e26ec9a4acd6ab6f67845de49db1972d8cbe1f4976df2fa3aeb23e6fe816bd68: Status 404 returned error can't find the container with id e26ec9a4acd6ab6f67845de49db1972d8cbe1f4976df2fa3aeb23e6fe816bd68
Sep 18 16:07:08.046478 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:08.046445    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69shv" event={"ID":"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b","Type":"ContainerStarted","Data":"d7cc99a87a213da5c1459542baf9d10bfb3777cf04ae4312299340aef61e810d"}
Sep 18 16:07:08.046619 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:08.046506    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69shv" event={"ID":"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b","Type":"ContainerStarted","Data":"e26ec9a4acd6ab6f67845de49db1972d8cbe1f4976df2fa3aeb23e6fe816bd68"}
Sep 18 16:07:09.051820 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:09.051781    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69shv" event={"ID":"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b","Type":"ContainerStarted","Data":"ce01ae5b40ff00401e13184aa421ad88c296a44ea9efc06e729f318720336581"}
Sep 18 16:07:10.820662 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.820625    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p"]
Sep 18 16:07:10.823835 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.823814    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p"
Sep 18 16:07:10.827828 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.827691    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 16:07:10.827828 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.827709    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 16:07:10.828108 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.828087    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-pfgg2\""
Sep 18 16:07:10.836451 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.836422    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p"]
Sep 18 16:07:10.931741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.931705    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-667f675df5-l9bfb"]
Sep 18 16:07:10.934783 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.934760    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:10.937350 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.937329    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 16:07:10.937977 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.937941    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 16:07:10.937977 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.937947    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bxqxp\""
Sep 18 16:07:10.938153 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.938106    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 16:07:10.943528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.943508    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 16:07:10.950704 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:10.950673    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-667f675df5-l9bfb"]
Sep 18 16:07:11.005423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.005386    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6dd55250-53ec-4e73-a50c-0fba9472f458-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-rpw7p\" (UID: \"6dd55250-53ec-4e73-a50c-0fba9472f458\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p"
Sep 18 16:07:11.005566 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.005447    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6dd55250-53ec-4e73-a50c-0fba9472f458-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-rpw7p\" (UID: \"6dd55250-53ec-4e73-a50c-0fba9472f458\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p"
Sep 18 16:07:11.060801 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.060763    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69shv" event={"ID":"a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b","Type":"ContainerStarted","Data":"019e627d77907f00236f55a583b464fe202b05e0a7795164bcc737ede7d6d37d"}
Sep 18 16:07:11.083459 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.083357    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-69shv" podStartSLOduration=17.831583541 podStartE2EDuration="20.083339004s" totalImagesPullingTime="2.251755463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:06:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:11.082395911 +0000 UTC m=+126.991176264" watchObservedRunningTime="2026-09-18 16:07:11.083339004 +0000 UTC m=+126.992119355"
Sep 18 16:07:11.105951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.105921    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-installation-pull-secrets\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.106134 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.105956    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-registry-certificates\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.106134 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.105977    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-ca-trust-extracted\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.106134 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.106020    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-registry-tls\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.106134 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.106128    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6dd55250-53ec-4e73-a50c-0fba9472f458-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-rpw7p\" (UID: \"6dd55250-53ec-4e73-a50c-0fba9472f458\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p"
Sep 18 16:07:11.106263 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.106159    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6dd55250-53ec-4e73-a50c-0fba9472f458-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-rpw7p\" (UID: \"6dd55250-53ec-4e73-a50c-0fba9472f458\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p"
Sep 18 16:07:11.106263 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.106177    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-bound-sa-token\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.106263 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.106206    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnt6g\" (UniqueName: \"kubernetes.io/projected/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-kube-api-access-pnt6g\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.106263 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.106230    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-trusted-ca\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.106418 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.106347    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-image-registry-private-configuration\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.106756 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.106736    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6dd55250-53ec-4e73-a50c-0fba9472f458-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-rpw7p\" (UID: \"6dd55250-53ec-4e73-a50c-0fba9472f458\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p"
Sep 18 16:07:11.108667 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.108649    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6dd55250-53ec-4e73-a50c-0fba9472f458-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-rpw7p\" (UID: \"6dd55250-53ec-4e73-a50c-0fba9472f458\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p"
Sep 18 16:07:11.133827 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.133801    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p"
Sep 18 16:07:11.207691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.207661    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-bound-sa-token\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.207691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.207696    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnt6g\" (UniqueName: \"kubernetes.io/projected/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-kube-api-access-pnt6g\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.208177 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.207739    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-trusted-ca\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.208177 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.207758    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-image-registry-private-configuration\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.208177 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.207779    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-installation-pull-secrets\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.208177 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.207809    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-registry-certificates\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.208177 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.207829    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-ca-trust-extracted\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.208177 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.207844    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-registry-tls\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.209075 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.209042    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-ca-trust-extracted\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.209205 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.209049    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-registry-certificates\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.209315 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.209296    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-trusted-ca\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.210558 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.210512    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-installation-pull-secrets\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.210832 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.210812    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-image-registry-private-configuration\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.211427 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.211404    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-registry-tls\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.219516 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.219487    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnt6g\" (UniqueName: \"kubernetes.io/projected/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-kube-api-access-pnt6g\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.219687 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.219671    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f835fe9f-0f6a-4ba6-b277-3897ab30a5f6-bound-sa-token\") pod \"image-registry-667f675df5-l9bfb\" (UID: \"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6\") " pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.244623 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.244595    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:11.404651 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.404576    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p"]
Sep 18 16:07:11.407533 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:07:11.407506    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd55250_53ec_4e73_a50c_0fba9472f458.slice/crio-55469971c44341899faf88dea9f7b4e0dc48d40b24b36ab79e3a210047495c1d WatchSource:0}: Error finding container 55469971c44341899faf88dea9f7b4e0dc48d40b24b36ab79e3a210047495c1d: Status 404 returned error can't find the container with id 55469971c44341899faf88dea9f7b4e0dc48d40b24b36ab79e3a210047495c1d
Sep 18 16:07:11.521198 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:11.521163    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-667f675df5-l9bfb"]
Sep 18 16:07:11.523742 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:07:11.523710    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf835fe9f_0f6a_4ba6_b277_3897ab30a5f6.slice/crio-ad5aeae9fb091a7bf75eebba98e4e4f21dcf6d92b962add8dab538aedcece51a WatchSource:0}: Error finding container ad5aeae9fb091a7bf75eebba98e4e4f21dcf6d92b962add8dab538aedcece51a: Status 404 returned error can't find the container with id ad5aeae9fb091a7bf75eebba98e4e4f21dcf6d92b962add8dab538aedcece51a
Sep 18 16:07:12.066612 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:12.066565    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p" event={"ID":"6dd55250-53ec-4e73-a50c-0fba9472f458","Type":"ContainerStarted","Data":"55469971c44341899faf88dea9f7b4e0dc48d40b24b36ab79e3a210047495c1d"}
Sep 18 16:07:12.068490 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:12.068445    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-667f675df5-l9bfb" event={"ID":"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6","Type":"ContainerStarted","Data":"ae79cd8e44b886864cbfc71edfbb3681dc40dd34089692975f8925608c198961"}
Sep 18 16:07:12.068641 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:12.068499    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-667f675df5-l9bfb" event={"ID":"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6","Type":"ContainerStarted","Data":"ad5aeae9fb091a7bf75eebba98e4e4f21dcf6d92b962add8dab538aedcece51a"}
Sep 18 16:07:12.068744 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:12.068724    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:12.089814 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:12.089745    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-667f675df5-l9bfb" podStartSLOduration=2.089721036 podStartE2EDuration="2.089721036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:12.089371498 +0000 UTC m=+127.998151868" watchObservedRunningTime="2026-09-18 16:07:12.089721036 +0000 UTC m=+127.998501388"
Sep 18 16:07:13.073448 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:13.073399    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p" event={"ID":"6dd55250-53ec-4e73-a50c-0fba9472f458","Type":"ContainerStarted","Data":"9004294f1feb5c36363547a66417ef632b52dd09d66f26ff87d34e894fb678ad"}
Sep 18 16:07:13.093258 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:13.093205    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-rpw7p" podStartSLOduration=1.9295917710000001 podStartE2EDuration="3.093190099s" totalImagesPullingTime="1.163598328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:13.091781363 +0000 UTC m=+129.000561714" watchObservedRunningTime="2026-09-18 16:07:13.093190099 +0000 UTC m=+129.001970451"
Sep 18 16:07:14.333529 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:14.333492    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:07:14.335913 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:14.335887    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f737909-4b95-481b-b092-0816b5be6f34-metrics-certs\") pod \"network-metrics-daemon-7h8rh\" (UID: \"1f737909-4b95-481b-b092-0816b5be6f34\") " pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:07:14.510312 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:14.510275    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89djg\""
Sep 18 16:07:14.518385 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:14.518365    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7h8rh"
Sep 18 16:07:14.789163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:14.789138    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7h8rh"]
Sep 18 16:07:14.791572 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:07:14.791538    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f737909_4b95_481b_b092_0816b5be6f34.slice/crio-afe89868b76d370308f83a4e5aa34fcf08174976f6abd11b13f7dc4cddc9fe10 WatchSource:0}: Error finding container afe89868b76d370308f83a4e5aa34fcf08174976f6abd11b13f7dc4cddc9fe10: Status 404 returned error can't find the container with id afe89868b76d370308f83a4e5aa34fcf08174976f6abd11b13f7dc4cddc9fe10
Sep 18 16:07:15.080458 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:15.080374    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7h8rh" event={"ID":"1f737909-4b95-481b-b092-0816b5be6f34","Type":"ContainerStarted","Data":"afe89868b76d370308f83a4e5aa34fcf08174976f6abd11b13f7dc4cddc9fe10"}
Sep 18 16:07:16.085482 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:16.085390    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7h8rh" event={"ID":"1f737909-4b95-481b-b092-0816b5be6f34","Type":"ContainerStarted","Data":"04f4c0f05f5ab0fe6288257feff6e7d02dc21c2ada0113eb5709a0351a1330db"}
Sep 18 16:07:16.085903 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:16.085487    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7h8rh" event={"ID":"1f737909-4b95-481b-b092-0816b5be6f34","Type":"ContainerStarted","Data":"65f4c2c122d25631815ca0baf32426d617f6cde7108f67f9a1c600d09fcf0da5"}
Sep 18 16:07:16.101952 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:16.101905    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7h8rh" podStartSLOduration=131.128535047 podStartE2EDuration="2m12.101887828s" totalImagesPullingTime="973.352781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:16.100339942 +0000 UTC m=+132.009120315" watchObservedRunningTime="2026-09-18 16:07:16.101887828 +0000 UTC m=+132.010668183"
Sep 18 16:07:19.701614 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.701576    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gbrnd"]
Sep 18 16:07:19.706596 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.706568    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.709351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.709325    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kgw4g\""
Sep 18 16:07:19.709666 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.709578    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 16:07:19.709843 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.709828    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 16:07:19.710216 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.710201    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 16:07:19.710294 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.710247    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 16:07:19.772671 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.772627    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-textfile\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.772671 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.772674    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-tls\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.772903 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.772706    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-sys\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.772903 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.772723    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7bmr\" (UniqueName: \"kubernetes.io/projected/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-kube-api-access-z7bmr\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.772903 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.772761    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.772903 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.772797    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.772903 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.772846    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-wtmp\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.772903 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.772892    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-root\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.773195 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.772932    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-metrics-client-ca\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.873495 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.873457    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-root\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.873708 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.873524    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-metrics-client-ca\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.873708 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.873562    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-textfile\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.873708 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.873571    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-root\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.873708 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.873588    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-tls\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.873708 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.873667    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-sys\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.873708 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:07:19.873699    2546 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 16:07:19.873708 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.873702    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7bmr\" (UniqueName: \"kubernetes.io/projected/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-kube-api-access-z7bmr\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.874070 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.873746    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.874070 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:07:19.873769    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-tls podName:3a9469a9-e638-4bfb-ae8e-09cff76a4e1c nodeName:}" failed. No retries permitted until 2026-09-18 16:07:20.373747705 +0000 UTC m=+136.282528050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-tls") pod "node-exporter-gbrnd" (UID: "3a9469a9-e638-4bfb-ae8e-09cff76a4e1c") : secret "node-exporter-tls" not found
Sep 18 16:07:19.874070 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.873803    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.874070 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.873854    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-wtmp\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.874070 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.873951    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-textfile\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.874070 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.874010    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-wtmp\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.874374 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.874333    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-metrics-client-ca\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.874427 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.874332    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-sys\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.874498 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.874478    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.876271 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.876244    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:19.886317 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:19.886287    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7bmr\" (UniqueName: \"kubernetes.io/projected/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-kube-api-access-z7bmr\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:20.377574 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:20.377537    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-tls\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:20.379867 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:20.379843    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3a9469a9-e638-4bfb-ae8e-09cff76a4e1c-node-exporter-tls\") pod \"node-exporter-gbrnd\" (UID: \"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c\") " pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:20.618633 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:20.618587    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gbrnd"
Sep 18 16:07:20.630839 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:07:20.630803    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9469a9_e638_4bfb_ae8e_09cff76a4e1c.slice/crio-f9f521c69c711ecfcde92880dff682a0998884105a2676f37e8e3a5298ca1496 WatchSource:0}: Error finding container f9f521c69c711ecfcde92880dff682a0998884105a2676f37e8e3a5298ca1496: Status 404 returned error can't find the container with id f9f521c69c711ecfcde92880dff682a0998884105a2676f37e8e3a5298ca1496
Sep 18 16:07:21.105733 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:21.105697    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gbrnd" event={"ID":"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c","Type":"ContainerStarted","Data":"f9f521c69c711ecfcde92880dff682a0998884105a2676f37e8e3a5298ca1496"}
Sep 18 16:07:22.110290 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:22.110261    2546 generic.go:356] "Generic (PLEG): container finished" podID="3a9469a9-e638-4bfb-ae8e-09cff76a4e1c" containerID="4d0fc17a5b606d3cf80108190bb8ab08de8b403497f58309098052d63633e79f" exitCode=0
Sep 18 16:07:22.110635 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:22.110311    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gbrnd" event={"ID":"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c","Type":"ContainerDied","Data":"4d0fc17a5b606d3cf80108190bb8ab08de8b403497f58309098052d63633e79f"}
Sep 18 16:07:23.116367 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:23.116325    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gbrnd" event={"ID":"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c","Type":"ContainerStarted","Data":"636390157f7b13e18e52bb866a5ed056abbb199be9773a59fa4d06f2907db0a6"}
Sep 18 16:07:23.116752 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:23.116395    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gbrnd" event={"ID":"3a9469a9-e638-4bfb-ae8e-09cff76a4e1c","Type":"ContainerStarted","Data":"5cb01770ea9df616a077bedc99534e912aa9307c4f1183e730ef683711c85f55"}
Sep 18 16:07:23.137122 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:23.137066    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gbrnd" podStartSLOduration=2.844042977 podStartE2EDuration="4.137049394s" totalImagesPullingTime="1.23360549s" totalInitContainerRuntime="59.400927ms" isStatefulPod=true podCreationTimestamp="2026-09-18 16:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:23.135516867 +0000 UTC m=+139.044297238" watchObservedRunningTime="2026-09-18 16:07:23.137049394 +0000 UTC m=+139.045829743"
Sep 18 16:07:23.990596 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:23.990560    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-798fb7c859-58tmk"]
Sep 18 16:07:23.993605 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:23.993589    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:23.996245 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:23.996209    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 16:07:23.997318 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:23.997291    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 16:07:23.997318 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:23.997310    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 18 16:07:23.997476 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:23.997375    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f1pbi65cltjq6\""
Sep 18 16:07:23.997476 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:23.997375    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vf5nr\""
Sep 18 16:07:23.997476 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:23.997406    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 16:07:24.002442 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.002073    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1ba48c5-1444-4082-aa66-d33328cf5882-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.002442 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.002147    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c1ba48c5-1444-4082-aa66-d33328cf5882-audit-log\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.002442 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.002177    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c1ba48c5-1444-4082-aa66-d33328cf5882-secret-metrics-server-tls\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.002442 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.002239    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c1ba48c5-1444-4082-aa66-d33328cf5882-metrics-server-audit-profiles\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.002442 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.002290    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c1ba48c5-1444-4082-aa66-d33328cf5882-secret-metrics-server-client-certs\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.002442 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.002318    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qgs\" (UniqueName: \"kubernetes.io/projected/c1ba48c5-1444-4082-aa66-d33328cf5882-kube-api-access-w7qgs\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.002442 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.002403    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c1ba48c5-1444-4082-aa66-d33328cf5882-client-ca-bundle\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.003571 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.003552    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-798fb7c859-58tmk"]
Sep 18 16:07:24.102998 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.102956    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c1ba48c5-1444-4082-aa66-d33328cf5882-secret-metrics-server-client-certs\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.102998 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.102994    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qgs\" (UniqueName: \"kubernetes.io/projected/c1ba48c5-1444-4082-aa66-d33328cf5882-kube-api-access-w7qgs\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.103243 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.103013    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c1ba48c5-1444-4082-aa66-d33328cf5882-client-ca-bundle\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.103243 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.103086    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1ba48c5-1444-4082-aa66-d33328cf5882-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.103243 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.103120    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c1ba48c5-1444-4082-aa66-d33328cf5882-audit-log\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.103243 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.103137    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c1ba48c5-1444-4082-aa66-d33328cf5882-secret-metrics-server-tls\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.103243 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.103162    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c1ba48c5-1444-4082-aa66-d33328cf5882-metrics-server-audit-profiles\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.103691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.103658    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c1ba48c5-1444-4082-aa66-d33328cf5882-audit-log\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.103971 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.103942    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1ba48c5-1444-4082-aa66-d33328cf5882-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.104153 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.104135    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c1ba48c5-1444-4082-aa66-d33328cf5882-metrics-server-audit-profiles\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.105570 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.105544    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c1ba48c5-1444-4082-aa66-d33328cf5882-secret-metrics-server-tls\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.105653 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.105604    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c1ba48c5-1444-4082-aa66-d33328cf5882-secret-metrics-server-client-certs\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.105755 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.105733    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c1ba48c5-1444-4082-aa66-d33328cf5882-client-ca-bundle\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.113795 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.113769    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qgs\" (UniqueName: \"kubernetes.io/projected/c1ba48c5-1444-4082-aa66-d33328cf5882-kube-api-access-w7qgs\") pod \"metrics-server-798fb7c859-58tmk\" (UID: \"c1ba48c5-1444-4082-aa66-d33328cf5882\") " pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.302738 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.302641    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:24.575010 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:24.574936    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-798fb7c859-58tmk"]
Sep 18 16:07:24.577955 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:07:24.577928    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ba48c5_1444_4082_aa66_d33328cf5882.slice/crio-29a158c2aa8bb07680ebd9f77462ef0c8a20551a9ca32ffbc11b6fd8b2c51f4b WatchSource:0}: Error finding container 29a158c2aa8bb07680ebd9f77462ef0c8a20551a9ca32ffbc11b6fd8b2c51f4b: Status 404 returned error can't find the container with id 29a158c2aa8bb07680ebd9f77462ef0c8a20551a9ca32ffbc11b6fd8b2c51f4b
Sep 18 16:07:25.125293 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.125253    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-798fb7c859-58tmk" event={"ID":"c1ba48c5-1444-4082-aa66-d33328cf5882","Type":"ContainerStarted","Data":"29a158c2aa8bb07680ebd9f77462ef0c8a20551a9ca32ffbc11b6fd8b2c51f4b"}
Sep 18 16:07:25.715169 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.715135    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c6db48c6b-9nbc5"]
Sep 18 16:07:25.718306 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.718280    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.721067 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.721039    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 16:07:25.721067 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.721054    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h4tf4\""
Sep 18 16:07:25.722273 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.722219    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 16:07:25.722452 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.722278    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 16:07:25.722517 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.722501    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 16:07:25.722954 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.722583    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 16:07:25.722954 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.722641    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 16:07:25.722954 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.722903    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 16:07:25.728268 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.728069    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 16:07:25.728787 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.728625    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c6db48c6b-9nbc5"]
Sep 18 16:07:25.817751 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.817719    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09430da0-01e5-4708-9261-2ea06ef44784-console-serving-cert\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.817951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.817780    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-console-config\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.817951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.817815    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-service-ca\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.817951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.817842    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09430da0-01e5-4708-9261-2ea06ef44784-console-oauth-config\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.817951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.817900    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-trusted-ca-bundle\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.817951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.817932    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bzkt\" (UniqueName: \"kubernetes.io/projected/09430da0-01e5-4708-9261-2ea06ef44784-kube-api-access-9bzkt\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.818323 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.817982    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-oauth-serving-cert\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.918396 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.918341    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-trusted-ca-bundle\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.918599 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.918501    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bzkt\" (UniqueName: \"kubernetes.io/projected/09430da0-01e5-4708-9261-2ea06ef44784-kube-api-access-9bzkt\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.918599 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.918553    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-oauth-serving-cert\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.918720 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.918628    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09430da0-01e5-4708-9261-2ea06ef44784-console-serving-cert\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.918720 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.918685    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-console-config\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.918897 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.918874    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-service-ca\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.918987 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.918926    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09430da0-01e5-4708-9261-2ea06ef44784-console-oauth-config\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.919440 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.919415    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-oauth-serving-cert\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.919440 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.919432    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-trusted-ca-bundle\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.919625 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.919415    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-console-config\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.919625 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.919584    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-service-ca\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.921568 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.921540    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09430da0-01e5-4708-9261-2ea06ef44784-console-serving-cert\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.921703 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.921686    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09430da0-01e5-4708-9261-2ea06ef44784-console-oauth-config\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:25.927245 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:25.927213    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bzkt\" (UniqueName: \"kubernetes.io/projected/09430da0-01e5-4708-9261-2ea06ef44784-kube-api-access-9bzkt\") pod \"console-c6db48c6b-9nbc5\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") " pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:26.031590 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:26.031502    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:26.631291 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:26.631218    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c6db48c6b-9nbc5"]
Sep 18 16:07:26.634798 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:07:26.634762    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09430da0_01e5_4708_9261_2ea06ef44784.slice/crio-b7bbb03cf9e01220ca700af4ae03a70fb0e043df083fe7d0abcc434fc439cb91 WatchSource:0}: Error finding container b7bbb03cf9e01220ca700af4ae03a70fb0e043df083fe7d0abcc434fc439cb91: Status 404 returned error can't find the container with id b7bbb03cf9e01220ca700af4ae03a70fb0e043df083fe7d0abcc434fc439cb91
Sep 18 16:07:27.133916 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:27.133884    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6db48c6b-9nbc5" event={"ID":"09430da0-01e5-4708-9261-2ea06ef44784","Type":"ContainerStarted","Data":"b7bbb03cf9e01220ca700af4ae03a70fb0e043df083fe7d0abcc434fc439cb91"}
Sep 18 16:07:27.135492 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:27.135464    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-798fb7c859-58tmk" event={"ID":"c1ba48c5-1444-4082-aa66-d33328cf5882","Type":"ContainerStarted","Data":"fd76dcc43a10f62334d51db227a8cc16e34a97f535bf4db9417bea68ab40d016"}
Sep 18 16:07:27.152760 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:27.152713    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-798fb7c859-58tmk" podStartSLOduration=2.319808518 podStartE2EDuration="4.152697949s" totalImagesPullingTime="1.832889431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:27.151449085 +0000 UTC m=+143.060229465" watchObservedRunningTime="2026-09-18 16:07:27.152697949 +0000 UTC m=+143.061478300"
Sep 18 16:07:31.153338 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:31.153301    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6db48c6b-9nbc5" event={"ID":"09430da0-01e5-4708-9261-2ea06ef44784","Type":"ContainerStarted","Data":"1c7b14fa860e0b29a8b9a19d965abeef186d7bb6fbf0a78c1662ec3732cf4b94"}
Sep 18 16:07:31.172168 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:31.172115    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c6db48c6b-9nbc5" podStartSLOduration=2.138825735 podStartE2EDuration="6.17209714s" totalImagesPullingTime="4.033271405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:31.171369917 +0000 UTC m=+147.080150274" watchObservedRunningTime="2026-09-18 16:07:31.17209714 +0000 UTC m=+147.080877493"
Sep 18 16:07:31.249861 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:31.249824    2546 patch_prober.go:28] interesting pod/image-registry-667f675df5-l9bfb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:07:31.250014 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:31.249888    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-667f675df5-l9bfb" podUID="f835fe9f-0f6a-4ba6-b277-3897ab30a5f6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:07:33.077958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.077922    2546 patch_prober.go:28] interesting pod/image-registry-667f675df5-l9bfb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:07:33.078359 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.077977    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-667f675df5-l9bfb" podUID="f835fe9f-0f6a-4ba6-b277-3897ab30a5f6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:07:33.341160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.341081    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-857c7b5875-wnzcg"]
Sep 18 16:07:33.344397 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.344379    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.364139 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.364108    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857c7b5875-wnzcg"]
Sep 18 16:07:33.387096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.387062    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-trusted-ca-bundle\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.387096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.387102    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-serving-cert\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.387275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.387124    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2rkt\" (UniqueName: \"kubernetes.io/projected/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-kube-api-access-z2rkt\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.387275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.387193    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-config\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.387275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.387249    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-oauth-config\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.387376 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.387277    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-service-ca\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.387376 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.387299    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-oauth-serving-cert\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.488427 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.488381    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-config\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.488618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.488451    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-oauth-config\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.488618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.488486    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-service-ca\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.488618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.488512    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-oauth-serving-cert\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.488766 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.488650    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-trusted-ca-bundle\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.488766 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.488691    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-serving-cert\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.488766 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.488716    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2rkt\" (UniqueName: \"kubernetes.io/projected/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-kube-api-access-z2rkt\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.489333 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.489308    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-service-ca\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.489469 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.489308    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-config\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.489469 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.489417    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-oauth-serving-cert\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.489798 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.489770    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-trusted-ca-bundle\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.491259 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.491238    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-oauth-config\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.491346 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.491242    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-serving-cert\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.499171 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.499141    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2rkt\" (UniqueName: \"kubernetes.io/projected/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-kube-api-access-z2rkt\") pod \"console-857c7b5875-wnzcg\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") " pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.652996 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.652890    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:33.923780 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:33.923755    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-857c7b5875-wnzcg"]
Sep 18 16:07:33.926267 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:07:33.926236    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d56a84_a5c1_4c07_896c_6d70f6b93f04.slice/crio-1d5cebb90fe929d666cca90b9cc99485f0da3f99fb1e0c8b1294761f8a483f75 WatchSource:0}: Error finding container 1d5cebb90fe929d666cca90b9cc99485f0da3f99fb1e0c8b1294761f8a483f75: Status 404 returned error can't find the container with id 1d5cebb90fe929d666cca90b9cc99485f0da3f99fb1e0c8b1294761f8a483f75
Sep 18 16:07:34.165820 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:34.165784    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857c7b5875-wnzcg" event={"ID":"d7d56a84-a5c1-4c07-896c-6d70f6b93f04","Type":"ContainerStarted","Data":"09fdef4011ddf7dacf4b1ff55ff6083086aacf120281be12ced5dc3d4dc3ca28"}
Sep 18 16:07:34.166207 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:34.165832    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857c7b5875-wnzcg" event={"ID":"d7d56a84-a5c1-4c07-896c-6d70f6b93f04","Type":"ContainerStarted","Data":"1d5cebb90fe929d666cca90b9cc99485f0da3f99fb1e0c8b1294761f8a483f75"}
Sep 18 16:07:34.186829 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:34.186735    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-857c7b5875-wnzcg" podStartSLOduration=1.18671856 podStartE2EDuration="1.18671856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:34.185404342 +0000 UTC m=+150.094184723" watchObservedRunningTime="2026-09-18 16:07:34.18671856 +0000 UTC m=+150.095498913"
Sep 18 16:07:36.032437 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:36.032401    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:36.032854 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:36.032691    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:07:36.033872 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:36.033851    2546 patch_prober.go:28] interesting pod/console-c6db48c6b-9nbc5 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused" start-of-body=
Sep 18 16:07:36.033955 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:36.033912    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-c6db48c6b-9nbc5" podUID="09430da0-01e5-4708-9261-2ea06ef44784" containerName="console" probeResult="failure" output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused"
Sep 18 16:07:40.430662 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:07:40.430625    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-c7lb2" podUID="ad2fa51d-439b-425a-af0e-05a48b372205"
Sep 18 16:07:41.190902 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:41.190869    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:07:41.249427 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:41.249390    2546 patch_prober.go:28] interesting pod/image-registry-667f675df5-l9bfb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:07:41.249560 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:41.249448    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-667f675df5-l9bfb" podUID="f835fe9f-0f6a-4ba6-b277-3897ab30a5f6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:07:43.078011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:43.077976    2546 patch_prober.go:28] interesting pod/image-registry-667f675df5-l9bfb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:07:43.078493 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:43.078054    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-667f675df5-l9bfb" podUID="f835fe9f-0f6a-4ba6-b277-3897ab30a5f6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:07:43.653814 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:43.653761    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:43.653814 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:43.653828    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:07:43.655436 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:43.655413    2546 patch_prober.go:28] interesting pod/console-857c7b5875-wnzcg container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused" start-of-body=
Sep 18 16:07:43.655525 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:43.655464    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-857c7b5875-wnzcg" podUID="d7d56a84-a5c1-4c07-896c-6d70f6b93f04" containerName="console" probeResult="failure" output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused"
Sep 18 16:07:44.302778 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:44.302731    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:44.303309 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:44.302813    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:07:45.291661 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:45.291618    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:07:45.293940 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:45.293909    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad2fa51d-439b-425a-af0e-05a48b372205-metrics-tls\") pod \"dns-default-c7lb2\" (UID: \"ad2fa51d-439b-425a-af0e-05a48b372205\") " pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:07:45.395428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:45.395391    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bgqnj\""
Sep 18 16:07:45.402576 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:45.402550    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:07:45.676502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:45.676463    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c7lb2"]
Sep 18 16:07:45.678746 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:07:45.678719    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2fa51d_439b_425a_af0e_05a48b372205.slice/crio-7c21ec3cf35ec4eb82bc26c8f565391b8993f14a11a6ca4dffcf1e829a846ec4 WatchSource:0}: Error finding container 7c21ec3cf35ec4eb82bc26c8f565391b8993f14a11a6ca4dffcf1e829a846ec4: Status 404 returned error can't find the container with id 7c21ec3cf35ec4eb82bc26c8f565391b8993f14a11a6ca4dffcf1e829a846ec4
Sep 18 16:07:46.032438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:46.032401    2546 patch_prober.go:28] interesting pod/console-c6db48c6b-9nbc5 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused" start-of-body=
Sep 18 16:07:46.032615 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:46.032459    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-c6db48c6b-9nbc5" podUID="09430da0-01e5-4708-9261-2ea06ef44784" containerName="console" probeResult="failure" output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused"
Sep 18 16:07:46.209917 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:46.209884    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c7lb2" event={"ID":"ad2fa51d-439b-425a-af0e-05a48b372205","Type":"ContainerStarted","Data":"7c21ec3cf35ec4eb82bc26c8f565391b8993f14a11a6ca4dffcf1e829a846ec4"}
Sep 18 16:07:48.219313 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:48.219279    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c7lb2" event={"ID":"ad2fa51d-439b-425a-af0e-05a48b372205","Type":"ContainerStarted","Data":"2c68c545c3d1d023bdad8a7a5b9a91203e18b225d51d29b3facdc521ad59a96b"}
Sep 18 16:07:48.219700 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:48.219334    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c7lb2" event={"ID":"ad2fa51d-439b-425a-af0e-05a48b372205","Type":"ContainerStarted","Data":"ab89f32566778fac8226e61ae3caaa93ae632ac568dd31c1c1b12299445fe1a5"}
Sep 18 16:07:48.219700 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:48.219410    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:07:48.239262 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:48.239207    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c7lb2" podStartSLOduration=129.421629863 podStartE2EDuration="2m11.239191579s" totalImagesPullingTime="1.817561716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:48.238469915 +0000 UTC m=+164.147250291" watchObservedRunningTime="2026-09-18 16:07:48.239191579 +0000 UTC m=+164.147971930"
Sep 18 16:07:51.248566 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:51.248530    2546 patch_prober.go:28] interesting pod/image-registry-667f675df5-l9bfb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:07:51.248939 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:51.248582    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-667f675df5-l9bfb" podUID="f835fe9f-0f6a-4ba6-b277-3897ab30a5f6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:07:51.248939 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:51.248625    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:07:51.249193 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:51.249162    2546 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"ae79cd8e44b886864cbfc71edfbb3681dc40dd34089692975f8925608c198961"} pod="openshift-image-registry/image-registry-667f675df5-l9bfb" containerMessage="Container registry failed liveness probe, will be restarted"
Sep 18 16:07:51.252683 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:51.252656    2546 patch_prober.go:28] interesting pod/image-registry-667f675df5-l9bfb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:07:51.252792 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:51.252695    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-667f675df5-l9bfb" podUID="f835fe9f-0f6a-4ba6-b277-3897ab30a5f6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:07:53.654081 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:53.654006    2546 patch_prober.go:28] interesting pod/console-857c7b5875-wnzcg container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused" start-of-body=
Sep 18 16:07:53.654473 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:53.654117    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-857c7b5875-wnzcg" podUID="d7d56a84-a5c1-4c07-896c-6d70f6b93f04" containerName="console" probeResult="failure" output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused"
Sep 18 16:07:56.031981 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:56.031941    2546 patch_prober.go:28] interesting pod/console-c6db48c6b-9nbc5 container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused" start-of-body=
Sep 18 16:07:56.032391 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:56.032000    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-c6db48c6b-9nbc5" podUID="09430da0-01e5-4708-9261-2ea06ef44784" containerName="console" probeResult="failure" output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection refused"
Sep 18 16:07:58.228168 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:07:58.228093    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c7lb2"
Sep 18 16:08:01.253548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:01.253510    2546 patch_prober.go:28] interesting pod/image-registry-667f675df5-l9bfb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 16:08:01.254003 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:01.253564    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-667f675df5-l9bfb" podUID="f835fe9f-0f6a-4ba6-b277-3897ab30a5f6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:08:03.657983 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:03.657946    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:08:03.661859 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:03.661830    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:08:03.722301 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:03.722271    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c6db48c6b-9nbc5"]
Sep 18 16:08:04.308765 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:04.308736    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:08:04.312691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:04.312665    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-798fb7c859-58tmk"
Sep 18 16:08:07.205048 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:07.204999    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c7lb2_ad2fa51d-439b-425a-af0e-05a48b372205/dns/0.log"
Sep 18 16:08:07.405546 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:07.405513    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c7lb2_ad2fa51d-439b-425a-af0e-05a48b372205/kube-rbac-proxy/0.log"
Sep 18 16:08:08.805421 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:08.805393    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2bl6_2023cf66-25c9-441a-b1b9-6892cf0b2746/dns-node-resolver/0.log"
Sep 18 16:08:09.012347 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:09.012319    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-667f675df5-l9bfb_f835fe9f-0f6a-4ba6-b277-3897ab30a5f6/registry/0.log"
Sep 18 16:08:09.604684 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:09.604654    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6gkbl_49ffc9f3-7c5f-48bc-8217-838ac6651979/node-ca/0.log"
Sep 18 16:08:10.005300 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:10.005269    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-847f74c87d-nnkct_fdf89ce0-2abf-4a51-a332-8c9c01121c6a/router/0.log"
Sep 18 16:08:10.604953 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:10.604923    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r7pf8_2de82e27-28f6-4ee2-bf84-74b92409a9b8/serve-healthcheck-canary/0.log"
Sep 18 16:08:11.253043 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:11.253005    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:08:14.319601 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:14.319515    2546 generic.go:356] "Generic (PLEG): container finished" podID="8e62eede-ea1b-415e-8549-3df93215459e" containerID="3cb91fd86824f74c5df2771e2af0257629209ab209887c473a880e2367c2d2c5" exitCode=0
Sep 18 16:08:14.319951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:14.319588    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7" event={"ID":"8e62eede-ea1b-415e-8549-3df93215459e","Type":"ContainerDied","Data":"3cb91fd86824f74c5df2771e2af0257629209ab209887c473a880e2367c2d2c5"}
Sep 18 16:08:14.319951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:14.319925    2546 scope.go:117] "RemoveContainer" containerID="3cb91fd86824f74c5df2771e2af0257629209ab209887c473a880e2367c2d2c5"
Sep 18 16:08:15.325533 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:15.325497    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7" event={"ID":"8e62eede-ea1b-415e-8549-3df93215459e","Type":"ContainerStarted","Data":"1eed94809c2cc96c8d5e9ea14d6e792498e4a77263e558dd988feb6fa7d9f564"}
Sep 18 16:08:16.267668 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:16.267618    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-667f675df5-l9bfb" podUID="f835fe9f-0f6a-4ba6-b277-3897ab30a5f6" containerName="registry" containerID="cri-o://ae79cd8e44b886864cbfc71edfbb3681dc40dd34089692975f8925608c198961" gracePeriod=30
Sep 18 16:08:18.339904 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:18.339869    2546 generic.go:356] "Generic (PLEG): container finished" podID="f835fe9f-0f6a-4ba6-b277-3897ab30a5f6" containerID="ae79cd8e44b886864cbfc71edfbb3681dc40dd34089692975f8925608c198961" exitCode=0
Sep 18 16:08:18.340378 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:18.339926    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-667f675df5-l9bfb" event={"ID":"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6","Type":"ContainerDied","Data":"ae79cd8e44b886864cbfc71edfbb3681dc40dd34089692975f8925608c198961"}
Sep 18 16:08:18.340378 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:18.339998    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-667f675df5-l9bfb" event={"ID":"f835fe9f-0f6a-4ba6-b277-3897ab30a5f6","Type":"ContainerStarted","Data":"fcacb23b6108ebeccb5788da8c7375cd5c4bc8e1a947aca96aa89f7b36faa457"}
Sep 18 16:08:18.340378 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:18.340018    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:08:28.742389 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:28.742332    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c6db48c6b-9nbc5" podUID="09430da0-01e5-4708-9261-2ea06ef44784" containerName="console" containerID="cri-o://1c7b14fa860e0b29a8b9a19d965abeef186d7bb6fbf0a78c1662ec3732cf4b94" gracePeriod=15
Sep 18 16:08:28.981672 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:28.981646    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:08:29.047210 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.047112    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bzkt\" (UniqueName: \"kubernetes.io/projected/09430da0-01e5-4708-9261-2ea06ef44784-kube-api-access-9bzkt\") pod \"09430da0-01e5-4708-9261-2ea06ef44784\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") "
Sep 18 16:08:29.047210 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.047168    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09430da0-01e5-4708-9261-2ea06ef44784-console-serving-cert\") pod \"09430da0-01e5-4708-9261-2ea06ef44784\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") "
Sep 18 16:08:29.047428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.047224    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-console-config\") pod \"09430da0-01e5-4708-9261-2ea06ef44784\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") "
Sep 18 16:08:29.047428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.047286    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-service-ca\") pod \"09430da0-01e5-4708-9261-2ea06ef44784\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") "
Sep 18 16:08:29.047428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.047313    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09430da0-01e5-4708-9261-2ea06ef44784-console-oauth-config\") pod \"09430da0-01e5-4708-9261-2ea06ef44784\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") "
Sep 18 16:08:29.047428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.047333    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-trusted-ca-bundle\") pod \"09430da0-01e5-4708-9261-2ea06ef44784\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") "
Sep 18 16:08:29.047428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.047371    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-oauth-serving-cert\") pod \"09430da0-01e5-4708-9261-2ea06ef44784\" (UID: \"09430da0-01e5-4708-9261-2ea06ef44784\") "
Sep 18 16:08:29.047722 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.047691    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-console-config" (OuterVolumeSpecName: "console-config") pod "09430da0-01e5-4708-9261-2ea06ef44784" (UID: "09430da0-01e5-4708-9261-2ea06ef44784"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:08:29.047851 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.047806    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-service-ca" (OuterVolumeSpecName: "service-ca") pod "09430da0-01e5-4708-9261-2ea06ef44784" (UID: "09430da0-01e5-4708-9261-2ea06ef44784"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:08:29.047924 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.047864    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "09430da0-01e5-4708-9261-2ea06ef44784" (UID: "09430da0-01e5-4708-9261-2ea06ef44784"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:08:29.048115 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.048093    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "09430da0-01e5-4708-9261-2ea06ef44784" (UID: "09430da0-01e5-4708-9261-2ea06ef44784"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:08:29.049453 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.049430    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09430da0-01e5-4708-9261-2ea06ef44784-kube-api-access-9bzkt" (OuterVolumeSpecName: "kube-api-access-9bzkt") pod "09430da0-01e5-4708-9261-2ea06ef44784" (UID: "09430da0-01e5-4708-9261-2ea06ef44784"). InnerVolumeSpecName "kube-api-access-9bzkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:08:29.049836 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.049812    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09430da0-01e5-4708-9261-2ea06ef44784-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "09430da0-01e5-4708-9261-2ea06ef44784" (UID: "09430da0-01e5-4708-9261-2ea06ef44784"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:08:29.049876 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.049830    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09430da0-01e5-4708-9261-2ea06ef44784-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "09430da0-01e5-4708-9261-2ea06ef44784" (UID: "09430da0-01e5-4708-9261-2ea06ef44784"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:08:29.148369 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.148336    2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-oauth-serving-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:08:29.148369 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.148366    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bzkt\" (UniqueName: \"kubernetes.io/projected/09430da0-01e5-4708-9261-2ea06ef44784-kube-api-access-9bzkt\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:08:29.148369 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.148376    2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/09430da0-01e5-4708-9261-2ea06ef44784-console-serving-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:08:29.148579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.148384    2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-console-config\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:08:29.148579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.148393    2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:08:29.148579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.148402    2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/09430da0-01e5-4708-9261-2ea06ef44784-console-oauth-config\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:08:29.148579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.148410    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09430da0-01e5-4708-9261-2ea06ef44784-trusted-ca-bundle\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:08:29.381786 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.381712    2546 generic.go:356] "Generic (PLEG): container finished" podID="09430da0-01e5-4708-9261-2ea06ef44784" containerID="1c7b14fa860e0b29a8b9a19d965abeef186d7bb6fbf0a78c1662ec3732cf4b94" exitCode=0
Sep 18 16:08:29.381786 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.381773    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6db48c6b-9nbc5"
Sep 18 16:08:29.381946 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.381797    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6db48c6b-9nbc5" event={"ID":"09430da0-01e5-4708-9261-2ea06ef44784","Type":"ContainerDied","Data":"1c7b14fa860e0b29a8b9a19d965abeef186d7bb6fbf0a78c1662ec3732cf4b94"}
Sep 18 16:08:29.381946 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.381871    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6db48c6b-9nbc5" event={"ID":"09430da0-01e5-4708-9261-2ea06ef44784","Type":"ContainerDied","Data":"b7bbb03cf9e01220ca700af4ae03a70fb0e043df083fe7d0abcc434fc439cb91"}
Sep 18 16:08:29.381946 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.381892    2546 scope.go:117] "RemoveContainer" containerID="1c7b14fa860e0b29a8b9a19d965abeef186d7bb6fbf0a78c1662ec3732cf4b94"
Sep 18 16:08:29.393245 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.393227    2546 scope.go:117] "RemoveContainer" containerID="1c7b14fa860e0b29a8b9a19d965abeef186d7bb6fbf0a78c1662ec3732cf4b94"
Sep 18 16:08:29.393496 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:08:29.393479    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c7b14fa860e0b29a8b9a19d965abeef186d7bb6fbf0a78c1662ec3732cf4b94\": container with ID starting with 1c7b14fa860e0b29a8b9a19d965abeef186d7bb6fbf0a78c1662ec3732cf4b94 not found: ID does not exist" containerID="1c7b14fa860e0b29a8b9a19d965abeef186d7bb6fbf0a78c1662ec3732cf4b94"
Sep 18 16:08:29.393543 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.393504    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7b14fa860e0b29a8b9a19d965abeef186d7bb6fbf0a78c1662ec3732cf4b94"} err="failed to get container status \"1c7b14fa860e0b29a8b9a19d965abeef186d7bb6fbf0a78c1662ec3732cf4b94\": rpc error: code = NotFound desc = could not find container \"1c7b14fa860e0b29a8b9a19d965abeef186d7bb6fbf0a78c1662ec3732cf4b94\": container with ID starting with 1c7b14fa860e0b29a8b9a19d965abeef186d7bb6fbf0a78c1662ec3732cf4b94 not found: ID does not exist"
Sep 18 16:08:29.405819 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.405793    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c6db48c6b-9nbc5"]
Sep 18 16:08:29.409571 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:29.409552    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c6db48c6b-9nbc5"]
Sep 18 16:08:30.601164 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:30.601133    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09430da0-01e5-4708-9261-2ea06ef44784" path="/var/lib/kubelet/pods/09430da0-01e5-4708-9261-2ea06ef44784/volumes"
Sep 18 16:08:39.348860 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:39.348829    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-667f675df5-l9bfb"
Sep 18 16:08:49.015528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:08:49.015495    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-857c7b5875-wnzcg"]
Sep 18 16:09:14.040715 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.040657    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-857c7b5875-wnzcg" podUID="d7d56a84-a5c1-4c07-896c-6d70f6b93f04" containerName="console" containerID="cri-o://09fdef4011ddf7dacf4b1ff55ff6083086aacf120281be12ced5dc3d4dc3ca28" gracePeriod=15
Sep 18 16:09:14.288281 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.288256    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:09:14.375847 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.375768    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-trusted-ca-bundle\") pod \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") "
Sep 18 16:09:14.375847 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.375812    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-serving-cert\") pod \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") "
Sep 18 16:09:14.376117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.375849    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-service-ca\") pod \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") "
Sep 18 16:09:14.376117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.375952    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-config\") pod \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") "
Sep 18 16:09:14.376117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.376012    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2rkt\" (UniqueName: \"kubernetes.io/projected/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-kube-api-access-z2rkt\") pod \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") "
Sep 18 16:09:14.376117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.376071    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-oauth-serving-cert\") pod \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") "
Sep 18 16:09:14.376117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.376109    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-oauth-config\") pod \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\" (UID: \"d7d56a84-a5c1-4c07-896c-6d70f6b93f04\") "
Sep 18 16:09:14.376367 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.376251    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-service-ca" (OuterVolumeSpecName: "service-ca") pod "d7d56a84-a5c1-4c07-896c-6d70f6b93f04" (UID: "d7d56a84-a5c1-4c07-896c-6d70f6b93f04"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:09:14.376367 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.376284    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d7d56a84-a5c1-4c07-896c-6d70f6b93f04" (UID: "d7d56a84-a5c1-4c07-896c-6d70f6b93f04"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:09:14.376469 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.376426    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-config" (OuterVolumeSpecName: "console-config") pod "d7d56a84-a5c1-4c07-896c-6d70f6b93f04" (UID: "d7d56a84-a5c1-4c07-896c-6d70f6b93f04"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:09:14.376517 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.376468    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-trusted-ca-bundle\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:09:14.376517 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.376487    2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:09:14.376517 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.376482    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d7d56a84-a5c1-4c07-896c-6d70f6b93f04" (UID: "d7d56a84-a5c1-4c07-896c-6d70f6b93f04"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:09:14.378160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.378139    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d7d56a84-a5c1-4c07-896c-6d70f6b93f04" (UID: "d7d56a84-a5c1-4c07-896c-6d70f6b93f04"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:09:14.378232 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.378180    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-kube-api-access-z2rkt" (OuterVolumeSpecName: "kube-api-access-z2rkt") pod "d7d56a84-a5c1-4c07-896c-6d70f6b93f04" (UID: "d7d56a84-a5c1-4c07-896c-6d70f6b93f04"). InnerVolumeSpecName "kube-api-access-z2rkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:09:14.378232 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.378222    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d7d56a84-a5c1-4c07-896c-6d70f6b93f04" (UID: "d7d56a84-a5c1-4c07-896c-6d70f6b93f04"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:09:14.477056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.477003    2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-config\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:09:14.477056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.477052    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2rkt\" (UniqueName: \"kubernetes.io/projected/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-kube-api-access-z2rkt\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:09:14.477056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.477063    2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-oauth-serving-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:09:14.477267 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.477072    2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-oauth-config\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:09:14.477267 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.477080    2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7d56a84-a5c1-4c07-896c-6d70f6b93f04-console-serving-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:09:14.539662 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.539631    2546 generic.go:356] "Generic (PLEG): container finished" podID="d7d56a84-a5c1-4c07-896c-6d70f6b93f04" containerID="09fdef4011ddf7dacf4b1ff55ff6083086aacf120281be12ced5dc3d4dc3ca28" exitCode=0
Sep 18 16:09:14.539810 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.539691    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-857c7b5875-wnzcg"
Sep 18 16:09:14.539810 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.539719    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857c7b5875-wnzcg" event={"ID":"d7d56a84-a5c1-4c07-896c-6d70f6b93f04","Type":"ContainerDied","Data":"09fdef4011ddf7dacf4b1ff55ff6083086aacf120281be12ced5dc3d4dc3ca28"}
Sep 18 16:09:14.539810 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.539797    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-857c7b5875-wnzcg" event={"ID":"d7d56a84-a5c1-4c07-896c-6d70f6b93f04","Type":"ContainerDied","Data":"1d5cebb90fe929d666cca90b9cc99485f0da3f99fb1e0c8b1294761f8a483f75"}
Sep 18 16:09:14.539932 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.539818    2546 scope.go:117] "RemoveContainer" containerID="09fdef4011ddf7dacf4b1ff55ff6083086aacf120281be12ced5dc3d4dc3ca28"
Sep 18 16:09:14.547922 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.547901    2546 scope.go:117] "RemoveContainer" containerID="09fdef4011ddf7dacf4b1ff55ff6083086aacf120281be12ced5dc3d4dc3ca28"
Sep 18 16:09:14.548286 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:09:14.548269    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09fdef4011ddf7dacf4b1ff55ff6083086aacf120281be12ced5dc3d4dc3ca28\": container with ID starting with 09fdef4011ddf7dacf4b1ff55ff6083086aacf120281be12ced5dc3d4dc3ca28 not found: ID does not exist" containerID="09fdef4011ddf7dacf4b1ff55ff6083086aacf120281be12ced5dc3d4dc3ca28"
Sep 18 16:09:14.548348 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.548294    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09fdef4011ddf7dacf4b1ff55ff6083086aacf120281be12ced5dc3d4dc3ca28"} err="failed to get container status \"09fdef4011ddf7dacf4b1ff55ff6083086aacf120281be12ced5dc3d4dc3ca28\": rpc error: code = NotFound desc = could not find container \"09fdef4011ddf7dacf4b1ff55ff6083086aacf120281be12ced5dc3d4dc3ca28\": container with ID starting with 09fdef4011ddf7dacf4b1ff55ff6083086aacf120281be12ced5dc3d4dc3ca28 not found: ID does not exist"
Sep 18 16:09:14.563405 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.563379    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-857c7b5875-wnzcg"]
Sep 18 16:09:14.571192 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.571172    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-857c7b5875-wnzcg"]
Sep 18 16:09:14.601266 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:09:14.601244    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7d56a84-a5c1-4c07-896c-6d70f6b93f04" path="/var/lib/kubelet/pods/d7d56a84-a5c1-4c07-896c-6d70f6b93f04/volumes"
Sep 18 16:10:04.477969 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:04.477942    2546 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 16:10:57.919927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.919849    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r"]
Sep 18 16:10:57.920330 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.920147    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09430da0-01e5-4708-9261-2ea06ef44784" containerName="console"
Sep 18 16:10:57.920330 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.920159    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="09430da0-01e5-4708-9261-2ea06ef44784" containerName="console"
Sep 18 16:10:57.920330 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.920166    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7d56a84-a5c1-4c07-896c-6d70f6b93f04" containerName="console"
Sep 18 16:10:57.920330 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.920171    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7d56a84-a5c1-4c07-896c-6d70f6b93f04" containerName="console"
Sep 18 16:10:57.920330 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.920233    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="09430da0-01e5-4708-9261-2ea06ef44784" containerName="console"
Sep 18 16:10:57.920330 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.920243    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7d56a84-a5c1-4c07-896c-6d70f6b93f04" containerName="console"
Sep 18 16:10:57.922979 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.922963    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r"
Sep 18 16:10:57.925375 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.925354    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 16:10:57.925510 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.925424    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 16:10:57.926491 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.926471    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8cf7l\""
Sep 18 16:10:57.934551 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.933208    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r"]
Sep 18 16:10:57.945919 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.945898    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d260854-98c0-4dd7-89a4-d9b9609d3203-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jcl8r\" (UID: \"5d260854-98c0-4dd7-89a4-d9b9609d3203\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r"
Sep 18 16:10:57.946006 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:57.945934    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kf4g\" (UniqueName: \"kubernetes.io/projected/5d260854-98c0-4dd7-89a4-d9b9609d3203-kube-api-access-2kf4g\") pod \"cert-manager-cainjector-559cdd8c87-jcl8r\" (UID: \"5d260854-98c0-4dd7-89a4-d9b9609d3203\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r"
Sep 18 16:10:58.047092 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:58.047063    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d260854-98c0-4dd7-89a4-d9b9609d3203-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jcl8r\" (UID: \"5d260854-98c0-4dd7-89a4-d9b9609d3203\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r"
Sep 18 16:10:58.047200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:58.047104    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kf4g\" (UniqueName: \"kubernetes.io/projected/5d260854-98c0-4dd7-89a4-d9b9609d3203-kube-api-access-2kf4g\") pod \"cert-manager-cainjector-559cdd8c87-jcl8r\" (UID: \"5d260854-98c0-4dd7-89a4-d9b9609d3203\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r"
Sep 18 16:10:58.056784 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:58.056760    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d260854-98c0-4dd7-89a4-d9b9609d3203-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-jcl8r\" (UID: \"5d260854-98c0-4dd7-89a4-d9b9609d3203\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r"
Sep 18 16:10:58.056855 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:58.056763    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kf4g\" (UniqueName: \"kubernetes.io/projected/5d260854-98c0-4dd7-89a4-d9b9609d3203-kube-api-access-2kf4g\") pod \"cert-manager-cainjector-559cdd8c87-jcl8r\" (UID: \"5d260854-98c0-4dd7-89a4-d9b9609d3203\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r"
Sep 18 16:10:58.251035 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:58.250998    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r"
Sep 18 16:10:58.520071 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:58.519981    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r"]
Sep 18 16:10:58.523172 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:10:58.523143    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d260854_98c0_4dd7_89a4_d9b9609d3203.slice/crio-46e29d1686154adc9a48b4b8ba0958ee1eef93f143788c02b7a4e574696cdf86 WatchSource:0}: Error finding container 46e29d1686154adc9a48b4b8ba0958ee1eef93f143788c02b7a4e574696cdf86: Status 404 returned error can't find the container with id 46e29d1686154adc9a48b4b8ba0958ee1eef93f143788c02b7a4e574696cdf86
Sep 18 16:10:58.525215 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:58.525194    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:10:58.900372 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:10:58.900288    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerStarted","Data":"46e29d1686154adc9a48b4b8ba0958ee1eef93f143788c02b7a4e574696cdf86"}
Sep 18 16:11:02.916380 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:02.916341    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerStarted","Data":"0bb3a863b4ce424950b1c22a41c6f615246bbf230234136b2d2e1e4256e5cc42"}
Sep 18 16:11:23.566426 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.566362    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podStartSLOduration=22.866956654 podStartE2EDuration="26.566344296s" totalImagesPullingTime="3.699387642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:10:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:11:02.93546285 +0000 UTC m=+358.844243253" watchObservedRunningTime="2026-09-18 16:11:23.566344296 +0000 UTC m=+379.475124648"
Sep 18 16:11:23.566934 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.566726    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"]
Sep 18 16:11:23.571410 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.571389    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.574582 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.574556    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mvfv2\""
Sep 18 16:11:23.575060 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.575040    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 16:11:23.575159 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.575143    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 16:11:23.575349 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.575326    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 16:11:23.575349 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.575342    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 16:11:23.587649 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.587624    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"]
Sep 18 16:11:23.640881 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.640844    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb9ee00b-5814-4425-a7e3-4062562574cc-tmp\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.641075 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.640896    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6qc6\" (UniqueName: \"kubernetes.io/projected/eb9ee00b-5814-4425-a7e3-4062562574cc-kube-api-access-x6qc6\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.641075 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.641000    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb9ee00b-5814-4425-a7e3-4062562574cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.641075 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.641059    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb9ee00b-5814-4425-a7e3-4062562574cc-manifests\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.641181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.641089    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb9ee00b-5814-4425-a7e3-4062562574cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.742294 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.742185    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb9ee00b-5814-4425-a7e3-4062562574cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.742514 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.742324    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb9ee00b-5814-4425-a7e3-4062562574cc-manifests\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.742514 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.742353    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb9ee00b-5814-4425-a7e3-4062562574cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.742514 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.742388    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb9ee00b-5814-4425-a7e3-4062562574cc-tmp\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.742514 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.742441    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6qc6\" (UniqueName: \"kubernetes.io/projected/eb9ee00b-5814-4425-a7e3-4062562574cc-kube-api-access-x6qc6\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.742692 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.742672    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/eb9ee00b-5814-4425-a7e3-4062562574cc-manifests\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.744595 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.744565    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/eb9ee00b-5814-4425-a7e3-4062562574cc-tmp\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.744728 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.744683    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/eb9ee00b-5814-4425-a7e3-4062562574cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.744870 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.744848    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb9ee00b-5814-4425-a7e3-4062562574cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.756221 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.756197    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6qc6\" (UniqueName: \"kubernetes.io/projected/eb9ee00b-5814-4425-a7e3-4062562574cc-kube-api-access-x6qc6\") pod \"opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws\" (UID: \"eb9ee00b-5814-4425-a7e3-4062562574cc\") " pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:23.881262 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:23.881162    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:24.160713 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:24.160629    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"]
Sep 18 16:11:24.164383 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:11:24.164355    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb9ee00b_5814_4425_a7e3_4062562574cc.slice/crio-478a7d1635212e2e4e287cccad5472228407b97b67cdd3c34dda4363bd2a61b6 WatchSource:0}: Error finding container 478a7d1635212e2e4e287cccad5472228407b97b67cdd3c34dda4363bd2a61b6: Status 404 returned error can't find the container with id 478a7d1635212e2e4e287cccad5472228407b97b67cdd3c34dda4363bd2a61b6
Sep 18 16:11:24.996952 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:24.996910    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerStarted","Data":"478a7d1635212e2e4e287cccad5472228407b97b67cdd3c34dda4363bd2a61b6"}
Sep 18 16:11:28.009918 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:28.009877    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerStarted","Data":"ef3a4bdaff547d7d2ba95660e8694c6a112c8f6df26fd74fd551a9f6af58b0b5"}
Sep 18 16:11:29.015639 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:29.015599    2546 generic.go:356] "Generic (PLEG): container finished" podID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerID="ef3a4bdaff547d7d2ba95660e8694c6a112c8f6df26fd74fd551a9f6af58b0b5" exitCode=0
Sep 18 16:11:29.016046 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:29.015662    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerDied","Data":"ef3a4bdaff547d7d2ba95660e8694c6a112c8f6df26fd74fd551a9f6af58b0b5"}
Sep 18 16:11:30.021825 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:30.021779    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerStarted","Data":"c28e358e7730fc0f82ecd3c76dbfd56ee4f554326adfdc0446501c801d293b3a"}
Sep 18 16:11:30.022389 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:30.021897    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:30.050577 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:30.050523    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podStartSLOduration=2.360522772 podStartE2EDuration="7.050506579s" totalImagesPullingTime="3.613019329s" totalInitContainerRuntime="1.076964478s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:11:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:11:30.048103266 +0000 UTC m=+385.956883632" watchObservedRunningTime="2026-09-18 16:11:30.050506579 +0000 UTC m=+385.959286931"
Sep 18 16:11:41.028426 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:41.028389    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:11:53.632383 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.632349    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"]
Sep 18 16:11:53.637080 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.637061    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"
Sep 18 16:11:53.641874 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.641846    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 16:11:53.642432 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.642413    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 16:11:53.664770 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.664747    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"]
Sep 18 16:11:53.781779 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.781746    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e823a1-2033-49b7-80be-18435d2aead9-tmp\") pod \"kube-auth-proxy-f97b6d9fd-tl7dn\" (UID: \"16e823a1-2033-49b7-80be-18435d2aead9\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"
Sep 18 16:11:53.781920 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.781790    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g78w\" (UniqueName: \"kubernetes.io/projected/16e823a1-2033-49b7-80be-18435d2aead9-kube-api-access-5g78w\") pod \"kube-auth-proxy-f97b6d9fd-tl7dn\" (UID: \"16e823a1-2033-49b7-80be-18435d2aead9\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"
Sep 18 16:11:53.781920 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.781817    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16e823a1-2033-49b7-80be-18435d2aead9-tls-certs\") pod \"kube-auth-proxy-f97b6d9fd-tl7dn\" (UID: \"16e823a1-2033-49b7-80be-18435d2aead9\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"
Sep 18 16:11:53.815853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.815827    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-klwkg"]
Sep 18 16:11:53.817813 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.817793    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:11:53.821016 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.820996    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 18 16:11:53.821169 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.821115    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-snrgd\""
Sep 18 16:11:53.837659 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.837638    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-klwkg"]
Sep 18 16:11:53.882712 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.882657    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g78w\" (UniqueName: \"kubernetes.io/projected/16e823a1-2033-49b7-80be-18435d2aead9-kube-api-access-5g78w\") pod \"kube-auth-proxy-f97b6d9fd-tl7dn\" (UID: \"16e823a1-2033-49b7-80be-18435d2aead9\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"
Sep 18 16:11:53.882712 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.882705    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16e823a1-2033-49b7-80be-18435d2aead9-tls-certs\") pod \"kube-auth-proxy-f97b6d9fd-tl7dn\" (UID: \"16e823a1-2033-49b7-80be-18435d2aead9\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"
Sep 18 16:11:53.882846 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.882775    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e823a1-2033-49b7-80be-18435d2aead9-tmp\") pod \"kube-auth-proxy-f97b6d9fd-tl7dn\" (UID: \"16e823a1-2033-49b7-80be-18435d2aead9\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"
Sep 18 16:11:53.882886 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:11:53.882860    2546 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 18 16:11:53.882937 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:11:53.882912    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16e823a1-2033-49b7-80be-18435d2aead9-tls-certs podName:16e823a1-2033-49b7-80be-18435d2aead9 nodeName:}" failed. No retries permitted until 2026-09-18 16:11:54.382897695 +0000 UTC m=+410.291678024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/16e823a1-2033-49b7-80be-18435d2aead9-tls-certs") pod "kube-auth-proxy-f97b6d9fd-tl7dn" (UID: "16e823a1-2033-49b7-80be-18435d2aead9") : secret "kube-auth-proxy-tls" not found
Sep 18 16:11:53.885003 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.884982    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/16e823a1-2033-49b7-80be-18435d2aead9-tmp\") pod \"kube-auth-proxy-f97b6d9fd-tl7dn\" (UID: \"16e823a1-2033-49b7-80be-18435d2aead9\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"
Sep 18 16:11:53.893627 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.893606    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g78w\" (UniqueName: \"kubernetes.io/projected/16e823a1-2033-49b7-80be-18435d2aead9-kube-api-access-5g78w\") pod \"kube-auth-proxy-f97b6d9fd-tl7dn\" (UID: \"16e823a1-2033-49b7-80be-18435d2aead9\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"
Sep 18 16:11:53.984105 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.984071    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs6pb\" (UniqueName: \"kubernetes.io/projected/4ce26bbe-bd40-4653-a16f-2b0c1c87604b-kube-api-access-zs6pb\") pod \"odh-model-controller-7998c67dc4-klwkg\" (UID: \"4ce26bbe-bd40-4653-a16f-2b0c1c87604b\") " pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:11:53.984281 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:53.984160    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ce26bbe-bd40-4653-a16f-2b0c1c87604b-cert\") pod \"odh-model-controller-7998c67dc4-klwkg\" (UID: \"4ce26bbe-bd40-4653-a16f-2b0c1c87604b\") " pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:11:54.084879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.084842    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ce26bbe-bd40-4653-a16f-2b0c1c87604b-cert\") pod \"odh-model-controller-7998c67dc4-klwkg\" (UID: \"4ce26bbe-bd40-4653-a16f-2b0c1c87604b\") " pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:11:54.085059 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.084898    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs6pb\" (UniqueName: \"kubernetes.io/projected/4ce26bbe-bd40-4653-a16f-2b0c1c87604b-kube-api-access-zs6pb\") pod \"odh-model-controller-7998c67dc4-klwkg\" (UID: \"4ce26bbe-bd40-4653-a16f-2b0c1c87604b\") " pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:11:54.085059 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:11:54.085003    2546 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 18 16:11:54.085148 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:11:54.085090    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ce26bbe-bd40-4653-a16f-2b0c1c87604b-cert podName:4ce26bbe-bd40-4653-a16f-2b0c1c87604b nodeName:}" failed. No retries permitted until 2026-09-18 16:11:54.585071355 +0000 UTC m=+410.493851685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4ce26bbe-bd40-4653-a16f-2b0c1c87604b-cert") pod "odh-model-controller-7998c67dc4-klwkg" (UID: "4ce26bbe-bd40-4653-a16f-2b0c1c87604b") : secret "odh-model-controller-webhook-cert" not found
Sep 18 16:11:54.098255 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.098225    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs6pb\" (UniqueName: \"kubernetes.io/projected/4ce26bbe-bd40-4653-a16f-2b0c1c87604b-kube-api-access-zs6pb\") pod \"odh-model-controller-7998c67dc4-klwkg\" (UID: \"4ce26bbe-bd40-4653-a16f-2b0c1c87604b\") " pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:11:54.387017 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.386985    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16e823a1-2033-49b7-80be-18435d2aead9-tls-certs\") pod \"kube-auth-proxy-f97b6d9fd-tl7dn\" (UID: \"16e823a1-2033-49b7-80be-18435d2aead9\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"
Sep 18 16:11:54.389371 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.389348    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16e823a1-2033-49b7-80be-18435d2aead9-tls-certs\") pod \"kube-auth-proxy-f97b6d9fd-tl7dn\" (UID: \"16e823a1-2033-49b7-80be-18435d2aead9\") " pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"
Sep 18 16:11:54.545466 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.545429    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"
Sep 18 16:11:54.589669 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.589605    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ce26bbe-bd40-4653-a16f-2b0c1c87604b-cert\") pod \"odh-model-controller-7998c67dc4-klwkg\" (UID: \"4ce26bbe-bd40-4653-a16f-2b0c1c87604b\") " pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:11:54.592802 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.592734    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ce26bbe-bd40-4653-a16f-2b0c1c87604b-cert\") pod \"odh-model-controller-7998c67dc4-klwkg\" (UID: \"4ce26bbe-bd40-4653-a16f-2b0c1c87604b\") " pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:11:54.727435 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.727400    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:11:54.729319 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.729293    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-g6r2f"]
Sep 18 16:11:54.731512 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.731487    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f"
Sep 18 16:11:54.734169 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.734006    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-phlgp\""
Sep 18 16:11:54.734488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.734453    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 18 16:11:54.747887 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.747855    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-g6r2f"]
Sep 18 16:11:54.829212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.828933    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn"]
Sep 18 16:11:54.892072 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.892018    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84711704-c750-4a14-bace-15805d228697-tls-certs\") pod \"model-serving-api-54d89bd5fc-g6r2f\" (UID: \"84711704-c750-4a14-bace-15805d228697\") " pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f"
Sep 18 16:11:54.892219 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.892144    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4lg\" (UniqueName: \"kubernetes.io/projected/84711704-c750-4a14-bace-15805d228697-kube-api-access-jk4lg\") pod \"model-serving-api-54d89bd5fc-g6r2f\" (UID: \"84711704-c750-4a14-bace-15805d228697\") " pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f"
Sep 18 16:11:54.992658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.992595    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84711704-c750-4a14-bace-15805d228697-tls-certs\") pod \"model-serving-api-54d89bd5fc-g6r2f\" (UID: \"84711704-c750-4a14-bace-15805d228697\") " pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f"
Sep 18 16:11:54.992777 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:54.992662    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4lg\" (UniqueName: \"kubernetes.io/projected/84711704-c750-4a14-bace-15805d228697-kube-api-access-jk4lg\") pod \"model-serving-api-54d89bd5fc-g6r2f\" (UID: \"84711704-c750-4a14-bace-15805d228697\") " pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f"
Sep 18 16:11:54.992777 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:11:54.992739    2546 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 18 16:11:54.992849 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:11:54.992801    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84711704-c750-4a14-bace-15805d228697-tls-certs podName:84711704-c750-4a14-bace-15805d228697 nodeName:}" failed. No retries permitted until 2026-09-18 16:11:55.492786008 +0000 UTC m=+411.401566337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/84711704-c750-4a14-bace-15805d228697-tls-certs") pod "model-serving-api-54d89bd5fc-g6r2f" (UID: "84711704-c750-4a14-bace-15805d228697") : secret "model-serving-api-tls" not found
Sep 18 16:11:55.002342 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.002311    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4lg\" (UniqueName: \"kubernetes.io/projected/84711704-c750-4a14-bace-15805d228697-kube-api-access-jk4lg\") pod \"model-serving-api-54d89bd5fc-g6r2f\" (UID: \"84711704-c750-4a14-bace-15805d228697\") " pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f"
Sep 18 16:11:55.009933 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.009911    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-klwkg"]
Sep 18 16:11:55.012277 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:11:55.012246    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce26bbe_bd40_4653_a16f_2b0c1c87604b.slice/crio-084cd00b985fdc7098ee0a1a47e781c3643b8fe6a22354ccff92f5543c7f6cf6 WatchSource:0}: Error finding container 084cd00b985fdc7098ee0a1a47e781c3643b8fe6a22354ccff92f5543c7f6cf6: Status 404 returned error can't find the container with id 084cd00b985fdc7098ee0a1a47e781c3643b8fe6a22354ccff92f5543c7f6cf6
Sep 18 16:11:55.122850 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.122816    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" event={"ID":"4ce26bbe-bd40-4653-a16f-2b0c1c87604b","Type":"ContainerStarted","Data":"084cd00b985fdc7098ee0a1a47e781c3643b8fe6a22354ccff92f5543c7f6cf6"}
Sep 18 16:11:55.124010 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.123987    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn" event={"ID":"16e823a1-2033-49b7-80be-18435d2aead9","Type":"ContainerStarted","Data":"e59ecc7b8421595f7a1cca11c1f99a4255c43df3b4a8c7ca6816e1cbc2f62d9c"}
Sep 18 16:11:55.243940 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.243864    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"]
Sep 18 16:11:55.246540 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.246523    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.249421 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.249397    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 16:11:55.249421 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.249411    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 16:11:55.249599 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.249448    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 16:11:55.249599 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.249506    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tt4rz\""
Sep 18 16:11:55.249745 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.249714    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 16:11:55.267902 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.267875    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"]
Sep 18 16:11:55.396749 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.396711    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.396925 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.396781    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.396925 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.396821    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.396925 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.396843    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.396925 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.396869    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.396925 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.396905    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpxcj\" (UniqueName: \"kubernetes.io/projected/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-kube-api-access-jpxcj\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.396925 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.396925    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.497710 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.497600    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.497710 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.497674    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.497915 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.497713    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.497915 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.497742    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.497915 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.497766    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.498115 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.497987    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpxcj\" (UniqueName: \"kubernetes.io/projected/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-kube-api-access-jpxcj\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.498115 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.498057    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.498216 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.498142    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84711704-c750-4a14-bace-15805d228697-tls-certs\") pod \"model-serving-api-54d89bd5fc-g6r2f\" (UID: \"84711704-c750-4a14-bace-15805d228697\") " pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f"
Sep 18 16:11:55.498529 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.498477    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.501446 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.501398    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.501446 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.501421    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84711704-c750-4a14-bace-15805d228697-tls-certs\") pod \"model-serving-api-54d89bd5fc-g6r2f\" (UID: \"84711704-c750-4a14-bace-15805d228697\") " pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f"
Sep 18 16:11:55.501680 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.501650    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.502459 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.502435    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.502590 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.502571    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.517413 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.517367    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.518351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.518324    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpxcj\" (UniqueName: \"kubernetes.io/projected/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-kube-api-access-jpxcj\") pod \"istiod-openshift-gateway-7b7995d5cd-d64dz\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.557945 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.557912    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:11:55.644266 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.644194    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f"
Sep 18 16:11:55.852408 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.852304    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"]
Sep 18 16:11:55.854528 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:11:55.854497    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc973d0b_a1ff_4ab2_bf81_5310ec08ae90.slice/crio-990fde9066bd1bad3b8134e287255700cf4d55e5f62c1a53fcdb28f33aec4d44 WatchSource:0}: Error finding container 990fde9066bd1bad3b8134e287255700cf4d55e5f62c1a53fcdb28f33aec4d44: Status 404 returned error can't find the container with id 990fde9066bd1bad3b8134e287255700cf4d55e5f62c1a53fcdb28f33aec4d44
Sep 18 16:11:55.936549 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:55.936464    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-g6r2f"]
Sep 18 16:11:55.941510 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:11:55.941477    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84711704_c750_4a14_bace_15805d228697.slice/crio-cf7c219c91129f0c30b62d2a9179dc81cdb103b8c256f70ff49a76e385821a76 WatchSource:0}: Error finding container cf7c219c91129f0c30b62d2a9179dc81cdb103b8c256f70ff49a76e385821a76: Status 404 returned error can't find the container with id cf7c219c91129f0c30b62d2a9179dc81cdb103b8c256f70ff49a76e385821a76
Sep 18 16:11:56.132148 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:56.132062    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f" event={"ID":"84711704-c750-4a14-bace-15805d228697","Type":"ContainerStarted","Data":"cf7c219c91129f0c30b62d2a9179dc81cdb103b8c256f70ff49a76e385821a76"}
Sep 18 16:11:56.134349 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:11:56.134318    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz" event={"ID":"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90","Type":"ContainerStarted","Data":"990fde9066bd1bad3b8134e287255700cf4d55e5f62c1a53fcdb28f33aec4d44"}
Sep 18 16:12:00.500912 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:00.500864    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:12:00.501221 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:00.500942    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:12:01.163754 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.163715    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" event={"ID":"4ce26bbe-bd40-4653-a16f-2b0c1c87604b","Type":"ContainerStarted","Data":"536ee85a2a92b0402144cad8b3453f6dd54c4a74f49bcf0a141ee2c7661b28cd"}
Sep 18 16:12:01.163938 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.163826    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:12:01.165353 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.165329    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn" event={"ID":"16e823a1-2033-49b7-80be-18435d2aead9","Type":"ContainerStarted","Data":"37fc225c6bbeb08e7aa54e30218f468b9f7559160b0a7031555ff647e4631d4e"}
Sep 18 16:12:01.166868 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.166846    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f" event={"ID":"84711704-c750-4a14-bace-15805d228697","Type":"ContainerStarted","Data":"ec4ac9c5b363ff871f329312d26c56a74134cfd4b3561dc53d650686e89d5679"}
Sep 18 16:12:01.166955 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.166916    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f"
Sep 18 16:12:01.168395 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.168373    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz" event={"ID":"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90","Type":"ContainerStarted","Data":"4cf689d216e82157d6cf865321bbc189cb97ee6b0d6ada8f17b6b84befb9e12b"}
Sep 18 16:12:01.168584 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.168568    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:12:01.170242 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.170220    2546 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-d64dz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 16:12:01.170325 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.170268    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz" podUID="cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:12:01.185573 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.185529    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" podStartSLOduration=2.83015995 podStartE2EDuration="8.185513011s" totalImagesPullingTime="5.355353061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:12:01.184171973 +0000 UTC m=+417.092952349" watchObservedRunningTime="2026-09-18 16:12:01.185513011 +0000 UTC m=+417.094293364"
Sep 18 16:12:01.236305 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.236241    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f" podStartSLOduration=2.8085146229999998 podStartE2EDuration="7.236219986s" totalImagesPullingTime="4.427705363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:12:01.233800056 +0000 UTC m=+417.142580432" watchObservedRunningTime="2026-09-18 16:12:01.236219986 +0000 UTC m=+417.145000341"
Sep 18 16:12:01.268742 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.268676    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz" podStartSLOduration=1.626589187 podStartE2EDuration="6.268635339s" totalImagesPullingTime="4.642046152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:11:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:12:01.267045125 +0000 UTC m=+417.175825503" watchObservedRunningTime="2026-09-18 16:12:01.268635339 +0000 UTC m=+417.177415694"
Sep 18 16:12:01.293604 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.293543    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-f97b6d9fd-tl7dn" podStartSLOduration=2.7652395260000002 podStartE2EDuration="8.293528808s" totalImagesPullingTime="5.528289282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:11:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:12:01.292802893 +0000 UTC m=+417.201583266" watchObservedRunningTime="2026-09-18 16:12:01.293528808 +0000 UTC m=+417.202309160"
Sep 18 16:12:01.667226 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.667192    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"]
Sep 18 16:12:01.669977 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.669958    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.672683 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.672663    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-b7zzg\""
Sep 18 16:12:01.691013 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.690987    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"]
Sep 18 16:12:01.854897 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.854871    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.854998 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.854921    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.854998 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.854939    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.855121 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.855012    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvpp\" (UniqueName: \"kubernetes.io/projected/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-kube-api-access-jcvpp\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.855121 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.855077    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.855121 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.855097    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.855121 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.855114    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.855245 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.855153    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.855245 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.855173    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.955801 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.955770    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.955801 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.955804    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.955996 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.955820    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.955996 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.955841    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.955996 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.955864    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.955996 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.955898    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.955996 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.955963    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.956275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.955990    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.956275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.956072    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvpp\" (UniqueName: \"kubernetes.io/projected/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-kube-api-access-jcvpp\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.956381 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.956273    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.956535 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.956502    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.956648 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.956540    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.956702 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.956652    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.956743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.956728    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.958669 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.958646    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.959124 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.959104    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.976404 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.976384    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.976962 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.976939    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvpp\" (UniqueName: \"kubernetes.io/projected/a9ca2260-8b26-43f3-9373-cf4ee2e25d9a-kube-api-access-jcvpp\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m\" (UID: \"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:01.980726 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:01.980705    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:02.174774 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:02.174748    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:12:02.268152 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:02.268126    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"]
Sep 18 16:12:02.270574 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:12:02.270546    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ca2260_8b26_43f3_9373_cf4ee2e25d9a.slice/crio-3d0e8ebf32f38c375523a1584a36dba9361c8bb1ca3121921c1d1f5210a5a1d9 WatchSource:0}: Error finding container 3d0e8ebf32f38c375523a1584a36dba9361c8bb1ca3121921c1d1f5210a5a1d9: Status 404 returned error can't find the container with id 3d0e8ebf32f38c375523a1584a36dba9361c8bb1ca3121921c1d1f5210a5a1d9
Sep 18 16:12:03.178992 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.178953    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m" event={"ID":"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a","Type":"ContainerStarted","Data":"3d0e8ebf32f38c375523a1584a36dba9361c8bb1ca3121921c1d1f5210a5a1d9"}
Sep 18 16:12:03.792983 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.792947    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"]
Sep 18 16:12:03.795346 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.795325    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:12:03.797962 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.797941    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 18 16:12:03.798187 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.798114    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jbsbs\""
Sep 18 16:12:03.819049 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.819006    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"]
Sep 18 16:12:03.826721 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.826679    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"]
Sep 18 16:12:03.829535 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.829514    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:12:03.833501 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.833479    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 18 16:12:03.833738 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.833719    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-r64dz\""
Sep 18 16:12:03.854636 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.854573    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"]
Sep 18 16:12:03.976368 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.976330    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm4d8\" (UniqueName: \"kubernetes.io/projected/d9b87cf9-1252-4eeb-bd58-194c81ea4a94-kube-api-access-wm4d8\") pod \"llmisvc-controller-manager-777c6bd7d5-8bfgv\" (UID: \"d9b87cf9-1252-4eeb-bd58-194c81ea4a94\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:12:03.976543 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.976380    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7-cert\") pod \"kserve-controller-manager-d6f4b65b8-pxtkv\" (UID: \"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:12:03.976543 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.976414    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ssb8\" (UniqueName: \"kubernetes.io/projected/f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7-kube-api-access-4ssb8\") pod \"kserve-controller-manager-d6f4b65b8-pxtkv\" (UID: \"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:12:03.976543 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:03.976435    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9b87cf9-1252-4eeb-bd58-194c81ea4a94-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8bfgv\" (UID: \"d9b87cf9-1252-4eeb-bd58-194c81ea4a94\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:12:04.077703 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:04.077625    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm4d8\" (UniqueName: \"kubernetes.io/projected/d9b87cf9-1252-4eeb-bd58-194c81ea4a94-kube-api-access-wm4d8\") pod \"llmisvc-controller-manager-777c6bd7d5-8bfgv\" (UID: \"d9b87cf9-1252-4eeb-bd58-194c81ea4a94\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:12:04.077703 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:04.077672    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7-cert\") pod \"kserve-controller-manager-d6f4b65b8-pxtkv\" (UID: \"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:12:04.077914 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:04.077707    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ssb8\" (UniqueName: \"kubernetes.io/projected/f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7-kube-api-access-4ssb8\") pod \"kserve-controller-manager-d6f4b65b8-pxtkv\" (UID: \"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:12:04.077914 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:04.077755    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9b87cf9-1252-4eeb-bd58-194c81ea4a94-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8bfgv\" (UID: \"d9b87cf9-1252-4eeb-bd58-194c81ea4a94\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:12:04.077914 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:12:04.077874    2546 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 18 16:12:04.078078 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:12:04.077936    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9b87cf9-1252-4eeb-bd58-194c81ea4a94-cert podName:d9b87cf9-1252-4eeb-bd58-194c81ea4a94 nodeName:}" failed. No retries permitted until 2026-09-18 16:12:04.577915868 +0000 UTC m=+420.486696203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d9b87cf9-1252-4eeb-bd58-194c81ea4a94-cert") pod "llmisvc-controller-manager-777c6bd7d5-8bfgv" (UID: "d9b87cf9-1252-4eeb-bd58-194c81ea4a94") : secret "llmisvc-webhook-server-cert" not found
Sep 18 16:12:04.080540 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:04.080513    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7-cert\") pod \"kserve-controller-manager-d6f4b65b8-pxtkv\" (UID: \"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:12:04.118253 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:04.118225    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm4d8\" (UniqueName: \"kubernetes.io/projected/d9b87cf9-1252-4eeb-bd58-194c81ea4a94-kube-api-access-wm4d8\") pod \"llmisvc-controller-manager-777c6bd7d5-8bfgv\" (UID: \"d9b87cf9-1252-4eeb-bd58-194c81ea4a94\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:12:04.120245 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:04.120223    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ssb8\" (UniqueName: \"kubernetes.io/projected/f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7-kube-api-access-4ssb8\") pod \"kserve-controller-manager-d6f4b65b8-pxtkv\" (UID: \"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:12:04.409150 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:04.409062    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:12:04.585908 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:04.585868    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9b87cf9-1252-4eeb-bd58-194c81ea4a94-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8bfgv\" (UID: \"d9b87cf9-1252-4eeb-bd58-194c81ea4a94\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:12:04.588673 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:04.588646    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d9b87cf9-1252-4eeb-bd58-194c81ea4a94-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8bfgv\" (UID: \"d9b87cf9-1252-4eeb-bd58-194c81ea4a94\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:12:04.745760 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:04.745724    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-r64dz\""
Sep 18 16:12:04.753259 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:04.753227    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:12:05.182948 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:05.180955    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:12:05.182948 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:05.181083    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:12:05.182948 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:05.181132    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:12:05.455329 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:05.455295    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"]
Sep 18 16:12:05.457192 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:12:05.457114    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd9b87cf9_1252_4eeb_bd58_194c81ea4a94.slice/crio-619470e412de54f8797238972ba66516c4dcfcb1ac84cdefa71c8d69dfd75dc8 WatchSource:0}: Error finding container 619470e412de54f8797238972ba66516c4dcfcb1ac84cdefa71c8d69dfd75dc8: Status 404 returned error can't find the container with id 619470e412de54f8797238972ba66516c4dcfcb1ac84cdefa71c8d69dfd75dc8
Sep 18 16:12:05.459427 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:05.459398    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"]
Sep 18 16:12:05.461854 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:12:05.461831    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bdde87_b8d3_4fa1_81db_9cd1668bbfb7.slice/crio-1ce3eb7e769e0756e92bdaa2fe07ecfd11a92d66cde8e291ea0caf01da901500 WatchSource:0}: Error finding container 1ce3eb7e769e0756e92bdaa2fe07ecfd11a92d66cde8e291ea0caf01da901500: Status 404 returned error can't find the container with id 1ce3eb7e769e0756e92bdaa2fe07ecfd11a92d66cde8e291ea0caf01da901500
Sep 18 16:12:05.912700 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:05.912610    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"]
Sep 18 16:12:05.913124 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:05.913092    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz" podUID="cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" containerName="discovery" containerID="cri-o://4cf689d216e82157d6cf865321bbc189cb97ee6b0d6ada8f17b6b84befb9e12b" gracePeriod=30
Sep 18 16:12:06.216008 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.215961    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m" event={"ID":"a9ca2260-8b26-43f3-9373-cf4ee2e25d9a","Type":"ContainerStarted","Data":"c2172477bfc98c51cf69466bc0a9d1e3d154c62d42ef9408f8d2aafbe79abc9a"}
Sep 18 16:12:06.217686 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.217662    2546 generic.go:356] "Generic (PLEG): container finished" podID="cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" containerID="4cf689d216e82157d6cf865321bbc189cb97ee6b0d6ada8f17b6b84befb9e12b" exitCode=0
Sep 18 16:12:06.217798 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.217699    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz" event={"ID":"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90","Type":"ContainerDied","Data":"4cf689d216e82157d6cf865321bbc189cb97ee6b0d6ada8f17b6b84befb9e12b"}
Sep 18 16:12:06.218936 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.218910    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" event={"ID":"d9b87cf9-1252-4eeb-bd58-194c81ea4a94","Type":"ContainerStarted","Data":"619470e412de54f8797238972ba66516c4dcfcb1ac84cdefa71c8d69dfd75dc8"}
Sep 18 16:12:06.220123 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.220101    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerStarted","Data":"1ce3eb7e769e0756e92bdaa2fe07ecfd11a92d66cde8e291ea0caf01da901500"}
Sep 18 16:12:06.257239 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.257197    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m" podStartSLOduration=2.349282057 podStartE2EDuration="5.257181676s" totalImagesPullingTime="2.907899619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:12:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:12:06.255793586 +0000 UTC m=+422.164573937" watchObservedRunningTime="2026-09-18 16:12:06.257181676 +0000 UTC m=+422.165962027"
Sep 18 16:12:06.272496 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.272467    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:12:06.297864 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.297825    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-token\") pod \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") "
Sep 18 16:12:06.298047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.297880    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-csr-ca-configmap\") pod \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") "
Sep 18 16:12:06.298047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.297918    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-cacerts\") pod \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") "
Sep 18 16:12:06.298047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.297951    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-csr-dns-cert\") pod \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") "
Sep 18 16:12:06.298047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.297992    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-kubeconfig\") pod \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") "
Sep 18 16:12:06.298257 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.298052    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-local-certs\") pod \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") "
Sep 18 16:12:06.298257 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.298115    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpxcj\" (UniqueName: \"kubernetes.io/projected/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-kube-api-access-jpxcj\") pod \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\" (UID: \"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90\") "
Sep 18 16:12:06.298376 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.298342    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" (UID: "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:12:06.299037 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.298984    2546 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-csr-ca-configmap\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:12:06.301278 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.301243    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" (UID: "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:12:06.302056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.301999    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" (UID: "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:12:06.302345 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.302311    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-cacerts" (OuterVolumeSpecName: "cacerts") pod "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" (UID: "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:12:06.302584 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.302556    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-kube-api-access-jpxcj" (OuterVolumeSpecName: "kube-api-access-jpxcj") pod "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" (UID: "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90"). InnerVolumeSpecName "kube-api-access-jpxcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:12:06.302667 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.302634    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-local-certs" (OuterVolumeSpecName: "local-certs") pod "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" (UID: "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:12:06.302848 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.302819    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" (UID: "cc973d0b-a1ff-4ab2-bf81-5310ec08ae90"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:12:06.400352 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.400262    2546 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-kubeconfig\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:12:06.400352 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.400289    2546 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-local-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:12:06.400352 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.400299    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpxcj\" (UniqueName: \"kubernetes.io/projected/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-kube-api-access-jpxcj\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:12:06.400352 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.400308    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-token\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:12:06.400352 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.400319    2546 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-cacerts\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:12:06.400352 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.400329    2546 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90-istio-csr-dns-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:12:06.981283 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.981247    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:06.987344 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:06.987315    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:07.226906 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:07.226650    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz" event={"ID":"cc973d0b-a1ff-4ab2-bf81-5310ec08ae90","Type":"ContainerDied","Data":"990fde9066bd1bad3b8134e287255700cf4d55e5f62c1a53fcdb28f33aec4d44"}
Sep 18 16:12:07.226906 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:07.226715    2546 scope.go:117] "RemoveContainer" containerID="4cf689d216e82157d6cf865321bbc189cb97ee6b0d6ada8f17b6b84befb9e12b"
Sep 18 16:12:07.226906 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:07.226861    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:07.228346 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:07.227204    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"
Sep 18 16:12:07.228346 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:07.228058    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m"
Sep 18 16:12:07.273698 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:07.273612    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"]
Sep 18 16:12:07.284469 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:07.284440    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-d64dz"]
Sep 18 16:12:08.234360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:08.234323    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerStarted","Data":"778bb1ef5585ff103e7dc27354d8781cc8f5383595cafc9fccc126ca3c7970dd"}
Sep 18 16:12:08.234824 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:08.234596    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:12:08.273890 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:08.273822    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podStartSLOduration=2.949511989 podStartE2EDuration="5.273800743s" totalImagesPullingTime="2.324288754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:12:08.268863061 +0000 UTC m=+424.177643415" watchObservedRunningTime="2026-09-18 16:12:08.273800743 +0000 UTC m=+424.182581096"
Sep 18 16:12:08.602184 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:08.602155    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" path="/var/lib/kubelet/pods/cc973d0b-a1ff-4ab2-bf81-5310ec08ae90/volumes"
Sep 18 16:12:09.239707 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:09.239666    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" event={"ID":"d9b87cf9-1252-4eeb-bd58-194c81ea4a94","Type":"ContainerStarted","Data":"38dcdd6f151f25ff14e35fca9d69d099ae915f1b50af9d604fb1bb1917144906"}
Sep 18 16:12:09.240200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:09.239972    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:12:09.273709 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:09.273655    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" podStartSLOduration=3.204193196 podStartE2EDuration="6.273637015s" totalImagesPullingTime="3.069443819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:12:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:12:09.272812963 +0000 UTC m=+425.181593338" watchObservedRunningTime="2026-09-18 16:12:09.273637015 +0000 UTC m=+425.182417367"
Sep 18 16:12:12.177490 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:12.177460    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:12:12.178703 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:12.178673    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-g6r2f"
Sep 18 16:12:26.991965 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:26.991880    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-595cbfdbf7-cx7g5"]
Sep 18 16:12:26.992411 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:26.992207    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" containerName="discovery"
Sep 18 16:12:26.992411 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:26.992220    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" containerName="discovery"
Sep 18 16:12:26.992411 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:26.992280    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc973d0b-a1ff-4ab2-bf81-5310ec08ae90" containerName="discovery"
Sep 18 16:12:26.995825 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:26.995807    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:26.998646 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:26.998621    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 16:12:26.998646 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:26.998627    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 16:12:26.999990 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:26.999970    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 16:12:27.000149 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:26.999984    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 16:12:27.000216 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:26.999984    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 16:12:27.000216 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.000000    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 16:12:27.000331 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.000012    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 16:12:27.000331 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.000047    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h4tf4\""
Sep 18 16:12:27.004267 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.004247    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 16:12:27.007503 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.007483    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595cbfdbf7-cx7g5"]
Sep 18 16:12:27.070346 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.070314    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-trusted-ca-bundle\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.070462 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.070352    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd7v9\" (UniqueName: \"kubernetes.io/projected/6143688a-e1f7-4af0-915f-649305788fbe-kube-api-access-bd7v9\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.070462 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.070424    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-console-config\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.070638 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.070466    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6143688a-e1f7-4af0-915f-649305788fbe-console-oauth-config\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.070638 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.070525    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-oauth-serving-cert\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.070638 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.070579    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-service-ca\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.070638 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.070616    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6143688a-e1f7-4af0-915f-649305788fbe-console-serving-cert\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.171270 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.171240    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-oauth-serving-cert\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.171384 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.171294    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-service-ca\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.171384 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.171316    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6143688a-e1f7-4af0-915f-649305788fbe-console-serving-cert\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.171384 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.171341    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-trusted-ca-bundle\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.171384 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.171365    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7v9\" (UniqueName: \"kubernetes.io/projected/6143688a-e1f7-4af0-915f-649305788fbe-kube-api-access-bd7v9\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.171555 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.171406    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-console-config\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.171555 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.171427    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6143688a-e1f7-4af0-915f-649305788fbe-console-oauth-config\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.172576 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.172546    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-oauth-serving-cert\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.172719 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.172695    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-service-ca\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.172999 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.172969    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-console-config\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.178869 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.178845    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-trusted-ca-bundle\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.178971 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.178845    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6143688a-e1f7-4af0-915f-649305788fbe-console-oauth-config\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.179082 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.179060    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6143688a-e1f7-4af0-915f-649305788fbe-console-serving-cert\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.181425 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.181393    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd7v9\" (UniqueName: \"kubernetes.io/projected/6143688a-e1f7-4af0-915f-649305788fbe-kube-api-access-bd7v9\") pod \"console-595cbfdbf7-cx7g5\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") " pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.305196 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.305137    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:27.588330 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:27.588307    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595cbfdbf7-cx7g5"]
Sep 18 16:12:27.590663 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:12:27.590629    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6143688a_e1f7_4af0_915f_649305788fbe.slice/crio-3c559423cebfe9ab3a1fb4ff7a022dfd5b3e28673617899f101e2909b8a0fd70 WatchSource:0}: Error finding container 3c559423cebfe9ab3a1fb4ff7a022dfd5b3e28673617899f101e2909b8a0fd70: Status 404 returned error can't find the container with id 3c559423cebfe9ab3a1fb4ff7a022dfd5b3e28673617899f101e2909b8a0fd70
Sep 18 16:12:28.320064 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:28.320014    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595cbfdbf7-cx7g5" event={"ID":"6143688a-e1f7-4af0-915f-649305788fbe","Type":"ContainerStarted","Data":"561c1aef3ab9d54bc7aa141804bdc4f2191087166f258dbb44a9a670ef421e5f"}
Sep 18 16:12:28.320485 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:28.320076    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595cbfdbf7-cx7g5" event={"ID":"6143688a-e1f7-4af0-915f-649305788fbe","Type":"ContainerStarted","Data":"3c559423cebfe9ab3a1fb4ff7a022dfd5b3e28673617899f101e2909b8a0fd70"}
Sep 18 16:12:28.344444 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:28.344393    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-595cbfdbf7-cx7g5" podStartSLOduration=2.344377907 podStartE2EDuration="2.344377907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:12:28.343521777 +0000 UTC m=+444.252302147" watchObservedRunningTime="2026-09-18 16:12:28.344377907 +0000 UTC m=+444.253158258"
Sep 18 16:12:37.306085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:37.306010    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:37.306500 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:37.306117    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:37.310901 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:37.310879    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:37.360699 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:37.360669    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:12:39.244940 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:39.244907    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:12:40.246856 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:12:40.246823    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:13:13.828368 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:13.828334    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-t7kmf"]
Sep 18 16:13:13.832950 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:13.832932    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:13:13.836494 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:13.836441    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-47677\""
Sep 18 16:13:13.836588 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:13.836451    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 16:13:13.837194 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:13.837179    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 16:13:13.846648 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:13.846627    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-t7kmf"]
Sep 18 16:13:13.928882 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:13.928856    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsdm5\" (UniqueName: \"kubernetes.io/projected/f7e45976-cc30-418c-96f4-70e0c667572e-kube-api-access-vsdm5\") pod \"authorino-operator-76b478bd78-t7kmf\" (UID: \"f7e45976-cc30-418c-96f4-70e0c667572e\") " pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:13:14.030178 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:14.030159    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsdm5\" (UniqueName: \"kubernetes.io/projected/f7e45976-cc30-418c-96f4-70e0c667572e-kube-api-access-vsdm5\") pod \"authorino-operator-76b478bd78-t7kmf\" (UID: \"f7e45976-cc30-418c-96f4-70e0c667572e\") " pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:13:14.047958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:14.047936    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsdm5\" (UniqueName: \"kubernetes.io/projected/f7e45976-cc30-418c-96f4-70e0c667572e-kube-api-access-vsdm5\") pod \"authorino-operator-76b478bd78-t7kmf\" (UID: \"f7e45976-cc30-418c-96f4-70e0c667572e\") " pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:13:14.142949 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:14.142896    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:13:14.432858 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:14.432786    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-t7kmf"]
Sep 18 16:13:14.435170 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:13:14.435144    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e45976_cc30_418c_96f4_70e0c667572e.slice/crio-0203cab49998338a46f0d591409f1c16aed5ef4ec10692f66ccf26bdda76835d WatchSource:0}: Error finding container 0203cab49998338a46f0d591409f1c16aed5ef4ec10692f66ccf26bdda76835d: Status 404 returned error can't find the container with id 0203cab49998338a46f0d591409f1c16aed5ef4ec10692f66ccf26bdda76835d
Sep 18 16:13:14.520279 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:14.520248    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" event={"ID":"f7e45976-cc30-418c-96f4-70e0c667572e","Type":"ContainerStarted","Data":"0203cab49998338a46f0d591409f1c16aed5ef4ec10692f66ccf26bdda76835d"}
Sep 18 16:13:16.024644 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:16.024614    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"]
Sep 18 16:13:16.028020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:16.027997    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:16.030837 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:16.030817    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tzx5z\""
Sep 18 16:13:16.041500 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:16.041477    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"]
Sep 18 16:13:16.046675 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:16.046653    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbfjs\" (UniqueName: \"kubernetes.io/projected/897afb16-d413-471a-9a63-12a7503736fe-kube-api-access-sbfjs\") pod \"limitador-operator-controller-manager-74495c69f8-5wwns\" (UID: \"897afb16-d413-471a-9a63-12a7503736fe\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:16.147821 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:16.147782    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfjs\" (UniqueName: \"kubernetes.io/projected/897afb16-d413-471a-9a63-12a7503736fe-kube-api-access-sbfjs\") pod \"limitador-operator-controller-manager-74495c69f8-5wwns\" (UID: \"897afb16-d413-471a-9a63-12a7503736fe\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:16.164926 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:16.164895    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbfjs\" (UniqueName: \"kubernetes.io/projected/897afb16-d413-471a-9a63-12a7503736fe-kube-api-access-sbfjs\") pod \"limitador-operator-controller-manager-74495c69f8-5wwns\" (UID: \"897afb16-d413-471a-9a63-12a7503736fe\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:16.338726 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:16.338644    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:17.237490 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:17.237408    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"]
Sep 18 16:13:17.239686 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:13:17.239660    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897afb16_d413_471a_9a63_12a7503736fe.slice/crio-4b61b51c0dcbd7cc907c0bb04ed5ae987ba83fc8d20fb729b946fb6799d25e6d WatchSource:0}: Error finding container 4b61b51c0dcbd7cc907c0bb04ed5ae987ba83fc8d20fb729b946fb6799d25e6d: Status 404 returned error can't find the container with id 4b61b51c0dcbd7cc907c0bb04ed5ae987ba83fc8d20fb729b946fb6799d25e6d
Sep 18 16:13:17.537105 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:17.537009    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" event={"ID":"f7e45976-cc30-418c-96f4-70e0c667572e","Type":"ContainerStarted","Data":"c9e1290130d2c65bb0fe3bcbf5b1c61eec051bc0cb3273a15f51a611fb97a9ff"}
Sep 18 16:13:17.537261 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:17.537158    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:13:17.538416 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:17.538391    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns" event={"ID":"897afb16-d413-471a-9a63-12a7503736fe","Type":"ContainerStarted","Data":"4b61b51c0dcbd7cc907c0bb04ed5ae987ba83fc8d20fb729b946fb6799d25e6d"}
Sep 18 16:13:17.559834 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:17.559790    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" podStartSLOduration=1.9700578499999999 podStartE2EDuration="4.559776975s" totalImagesPullingTime="2.589719125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:13:17.55730748 +0000 UTC m=+493.466087868" watchObservedRunningTime="2026-09-18 16:13:17.559776975 +0000 UTC m=+493.468557327"
Sep 18 16:13:19.551821 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:19.551783    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns" event={"ID":"897afb16-d413-471a-9a63-12a7503736fe","Type":"ContainerStarted","Data":"d1fb58ebf3de3878be2759728cd9879cbc21c3836c4ead7f3208c5d53abd8bcf"}
Sep 18 16:13:19.552236 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:19.552054    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:19.577093 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:19.577043    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns" podStartSLOduration=2.025408859 podStartE2EDuration="3.577014195s" totalImagesPullingTime="1.551605336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:13:19.575657177 +0000 UTC m=+495.484437530" watchObservedRunningTime="2026-09-18 16:13:19.577014195 +0000 UTC m=+495.485794547"
Sep 18 16:13:24.161934 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:24.161899    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"]
Sep 18 16:13:24.165332 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:24.165316    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:24.168465 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:24.168445    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jdlq5\""
Sep 18 16:13:24.180596 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:24.180573    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"]
Sep 18 16:13:24.210622 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:24.210597    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9198458-9a72-48e5-8e65-0f1eeff11c5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" (UID: \"f9198458-9a72-48e5-8e65-0f1eeff11c5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:24.210729 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:24.210643    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxsx\" (UniqueName: \"kubernetes.io/projected/f9198458-9a72-48e5-8e65-0f1eeff11c5a-kube-api-access-qgxsx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" (UID: \"f9198458-9a72-48e5-8e65-0f1eeff11c5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:24.311913 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:24.311885    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxsx\" (UniqueName: \"kubernetes.io/projected/f9198458-9a72-48e5-8e65-0f1eeff11c5a-kube-api-access-qgxsx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" (UID: \"f9198458-9a72-48e5-8e65-0f1eeff11c5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:24.312062 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:24.311954    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9198458-9a72-48e5-8e65-0f1eeff11c5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" (UID: \"f9198458-9a72-48e5-8e65-0f1eeff11c5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:24.312293 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:24.312273    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9198458-9a72-48e5-8e65-0f1eeff11c5a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" (UID: \"f9198458-9a72-48e5-8e65-0f1eeff11c5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:24.322402 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:24.322379    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxsx\" (UniqueName: \"kubernetes.io/projected/f9198458-9a72-48e5-8e65-0f1eeff11c5a-kube-api-access-qgxsx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" (UID: \"f9198458-9a72-48e5-8e65-0f1eeff11c5a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:24.475476 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:24.475449    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:24.749375 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:24.749284    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"]
Sep 18 16:13:24.753570 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:13:24.753543    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9198458_9a72_48e5_8e65_0f1eeff11c5a.slice/crio-9e714896119ccb0af5a35a961f1122f37e1cdc62177f719cf500e0699b0bdd55 WatchSource:0}: Error finding container 9e714896119ccb0af5a35a961f1122f37e1cdc62177f719cf500e0699b0bdd55: Status 404 returned error can't find the container with id 9e714896119ccb0af5a35a961f1122f37e1cdc62177f719cf500e0699b0bdd55
Sep 18 16:13:25.579936 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:25.579903    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp" event={"ID":"f9198458-9a72-48e5-8e65-0f1eeff11c5a","Type":"ContainerStarted","Data":"9e714896119ccb0af5a35a961f1122f37e1cdc62177f719cf500e0699b0bdd55"}
Sep 18 16:13:28.548426 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:28.548394    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:13:29.601420 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:29.601384    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp" event={"ID":"f9198458-9a72-48e5-8e65-0f1eeff11c5a","Type":"ContainerStarted","Data":"d25ee01038cfc8807cb713611eda06508d566d142473104d7a18c8e94f1eb284"}
Sep 18 16:13:29.601818 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:29.601513    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:29.626983 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:29.626943    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp" podStartSLOduration=1.7739336460000001 podStartE2EDuration="5.626928723s" totalImagesPullingTime="3.852995077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:13:29.625310308 +0000 UTC m=+505.534090685" watchObservedRunningTime="2026-09-18 16:13:29.626928723 +0000 UTC m=+505.535709076"
Sep 18 16:13:30.558521 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:30.558487    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:40.608464 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:40.608436    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:42.910797 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:42.910763    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"]
Sep 18 16:13:42.961397 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:42.961367    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"]
Sep 18 16:13:42.961542 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:42.961457    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:43.049789 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.049760    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"]
Sep 18 16:13:43.050000 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.049975    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-lvvcs], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc" podUID="fa5bac16-0d0d-418b-9866-223a843f69a2"
Sep 18 16:13:43.057993 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.057957    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"]
Sep 18 16:13:43.058175 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.058152    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" containerName="manager" containerID="cri-o://d25ee01038cfc8807cb713611eda06508d566d142473104d7a18c8e94f1eb284" gracePeriod=2
Sep 18 16:13:43.059927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.059902    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa5bac16-0d0d-418b-9866-223a843f69a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kffhc\" (UID: \"fa5bac16-0d0d-418b-9866-223a843f69a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:43.060041 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.059940    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvvcs\" (UniqueName: \"kubernetes.io/projected/fa5bac16-0d0d-418b-9866-223a843f69a2-kube-api-access-lvvcs\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kffhc\" (UID: \"fa5bac16-0d0d-418b-9866-223a843f69a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:43.074933 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.074906    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"]
Sep 18 16:13:43.076354 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.076322    2546 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"f9198458-9a72-48e5-8e65-0f1eeff11c5a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-t4msp\": pods \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:43.096167 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.096146    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"]
Sep 18 16:13:43.097504 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.097484    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"]
Sep 18 16:13:43.097786 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.097774    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" containerName="manager"
Sep 18 16:13:43.097829 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.097787    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" containerName="manager"
Sep 18 16:13:43.097867 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.097851    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" containerName="manager"
Sep 18 16:13:43.119130 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.119101    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"]
Sep 18 16:13:43.119285 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.119215    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"
Sep 18 16:13:43.124128 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.124100    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:43.136113 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.136095    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"]
Sep 18 16:13:43.136205 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.136134    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"]
Sep 18 16:13:43.136205 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.136171    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"]
Sep 18 16:13:43.136205 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.136198    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"]
Sep 18 16:13:43.136365 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.136243    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:13:43.136416 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.136379    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns" podUID="897afb16-d413-471a-9a63-12a7503736fe" containerName="manager" containerID="cri-o://d1fb58ebf3de3878be2759728cd9879cbc21c3836c4ead7f3208c5d53abd8bcf" gracePeriod=2
Sep 18 16:13:43.142597 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.142099    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"]
Sep 18 16:13:43.159280 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.159260    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"]
Sep 18 16:13:43.159618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.159603    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="897afb16-d413-471a-9a63-12a7503736fe" containerName="manager"
Sep 18 16:13:43.159692 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.159622    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="897afb16-d413-471a-9a63-12a7503736fe" containerName="manager"
Sep 18 16:13:43.159749 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.159700    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="897afb16-d413-471a-9a63-12a7503736fe" containerName="manager"
Sep 18 16:13:43.160962 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.160903    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa5bac16-0d0d-418b-9866-223a843f69a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kffhc\" (UID: \"fa5bac16-0d0d-418b-9866-223a843f69a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:43.160962 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.160945    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvvcs\" (UniqueName: \"kubernetes.io/projected/fa5bac16-0d0d-418b-9866-223a843f69a2-kube-api-access-lvvcs\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kffhc\" (UID: \"fa5bac16-0d0d-418b-9866-223a843f69a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:43.161136 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.160975    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8db8b0ba-8ed6-4015-8b68-3b133143fdc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-56jkq\" (UID: \"8db8b0ba-8ed6-4015-8b68-3b133143fdc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"
Sep 18 16:13:43.161136 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.161008    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779gz\" (UniqueName: \"kubernetes.io/projected/8db8b0ba-8ed6-4015-8b68-3b133143fdc3-kube-api-access-779gz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-56jkq\" (UID: \"8db8b0ba-8ed6-4015-8b68-3b133143fdc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"
Sep 18 16:13:43.161524 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.161502    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa5bac16-0d0d-418b-9866-223a843f69a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-kffhc\" (UID: \"fa5bac16-0d0d-418b-9866-223a843f69a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:43.165379 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.165352    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:43.170507 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.170477    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5wwns\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="897afb16-d413-471a-9a63-12a7503736fe" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:43.170949 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.170927    2546 projected.go:196] Error preparing data for projected volume kube-api-access-lvvcs for pod kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc: failed to fetch token: pod "kuadrant-operator-controller-manager-7978cbdfbc-kffhc" not found
Sep 18 16:13:43.171159 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.171016    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa5bac16-0d0d-418b-9866-223a843f69a2-kube-api-access-lvvcs podName:fa5bac16-0d0d-418b-9866-223a843f69a2 nodeName:}" failed. No retries permitted until 2026-09-18 16:13:43.67099437 +0000 UTC m=+519.579774720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lvvcs" (UniqueName: "kubernetes.io/projected/fa5bac16-0d0d-418b-9866-223a843f69a2-kube-api-access-lvvcs") pod "kuadrant-operator-controller-manager-7978cbdfbc-kffhc" (UID: "fa5bac16-0d0d-418b-9866-223a843f69a2") : failed to fetch token: pod "kuadrant-operator-controller-manager-7978cbdfbc-kffhc" not found
Sep 18 16:13:43.180786 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.180763    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"]
Sep 18 16:13:43.180895 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.180876    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:13:43.199453 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.199421    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:43.201519 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.201487    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5wwns\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="897afb16-d413-471a-9a63-12a7503736fe" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:43.261737 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.261704    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzwn\" (UniqueName: \"kubernetes.io/projected/1c53eeff-c125-49c6-ab26-182dd935f398-kube-api-access-xxzwn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l7p9n\" (UID: \"1c53eeff-c125-49c6-ab26-182dd935f398\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:13:43.261898 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.261841    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cg42\" (UniqueName: \"kubernetes.io/projected/61d67412-30af-4fb0-adc0-5251f88b7a41-kube-api-access-2cg42\") pod \"limitador-operator-controller-manager-74495c69f8-dbsdv\" (UID: \"61d67412-30af-4fb0-adc0-5251f88b7a41\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:13:43.261949 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.261908    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8db8b0ba-8ed6-4015-8b68-3b133143fdc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-56jkq\" (UID: \"8db8b0ba-8ed6-4015-8b68-3b133143fdc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"
Sep 18 16:13:43.261949 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.261941    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c53eeff-c125-49c6-ab26-182dd935f398-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l7p9n\" (UID: \"1c53eeff-c125-49c6-ab26-182dd935f398\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:13:43.262019 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.261977    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779gz\" (UniqueName: \"kubernetes.io/projected/8db8b0ba-8ed6-4015-8b68-3b133143fdc3-kube-api-access-779gz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-56jkq\" (UID: \"8db8b0ba-8ed6-4015-8b68-3b133143fdc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"
Sep 18 16:13:43.262275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.262258    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8db8b0ba-8ed6-4015-8b68-3b133143fdc3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-56jkq\" (UID: \"8db8b0ba-8ed6-4015-8b68-3b133143fdc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"
Sep 18 16:13:43.275231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.275207    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-779gz\" (UniqueName: \"kubernetes.io/projected/8db8b0ba-8ed6-4015-8b68-3b133143fdc3-kube-api-access-779gz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-56jkq\" (UID: \"8db8b0ba-8ed6-4015-8b68-3b133143fdc3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"
Sep 18 16:13:43.337436 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.337411    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:43.339993 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.339958    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:43.342195 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.342163    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5wwns\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="897afb16-d413-471a-9a63-12a7503736fe" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:43.363329 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.363304    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cg42\" (UniqueName: \"kubernetes.io/projected/61d67412-30af-4fb0-adc0-5251f88b7a41-kube-api-access-2cg42\") pod \"limitador-operator-controller-manager-74495c69f8-dbsdv\" (UID: \"61d67412-30af-4fb0-adc0-5251f88b7a41\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:13:43.363461 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.363446    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c53eeff-c125-49c6-ab26-182dd935f398-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l7p9n\" (UID: \"1c53eeff-c125-49c6-ab26-182dd935f398\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:13:43.363531 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.363511    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzwn\" (UniqueName: \"kubernetes.io/projected/1c53eeff-c125-49c6-ab26-182dd935f398-kube-api-access-xxzwn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l7p9n\" (UID: \"1c53eeff-c125-49c6-ab26-182dd935f398\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:13:43.363869 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.363844    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c53eeff-c125-49c6-ab26-182dd935f398-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l7p9n\" (UID: \"1c53eeff-c125-49c6-ab26-182dd935f398\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:13:43.373622 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.373603    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzwn\" (UniqueName: \"kubernetes.io/projected/1c53eeff-c125-49c6-ab26-182dd935f398-kube-api-access-xxzwn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l7p9n\" (UID: \"1c53eeff-c125-49c6-ab26-182dd935f398\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:13:43.373992 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.373966    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cg42\" (UniqueName: \"kubernetes.io/projected/61d67412-30af-4fb0-adc0-5251f88b7a41-kube-api-access-2cg42\") pod \"limitador-operator-controller-manager-74495c69f8-dbsdv\" (UID: \"61d67412-30af-4fb0-adc0-5251f88b7a41\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:13:43.401813 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.401794    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:43.404181 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.404158    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:43.406238 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.406207    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5wwns\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="897afb16-d413-471a-9a63-12a7503736fe" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:43.441118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.441075    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"
Sep 18 16:13:43.450223 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.450195    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:13:43.464272 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.464252    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgxsx\" (UniqueName: \"kubernetes.io/projected/f9198458-9a72-48e5-8e65-0f1eeff11c5a-kube-api-access-qgxsx\") pod \"f9198458-9a72-48e5-8e65-0f1eeff11c5a\" (UID: \"f9198458-9a72-48e5-8e65-0f1eeff11c5a\") "
Sep 18 16:13:43.464368 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.464352    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9198458-9a72-48e5-8e65-0f1eeff11c5a-extensions-socket-volume\") pod \"f9198458-9a72-48e5-8e65-0f1eeff11c5a\" (UID: \"f9198458-9a72-48e5-8e65-0f1eeff11c5a\") "
Sep 18 16:13:43.464455 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.464443    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbfjs\" (UniqueName: \"kubernetes.io/projected/897afb16-d413-471a-9a63-12a7503736fe-kube-api-access-sbfjs\") pod \"897afb16-d413-471a-9a63-12a7503736fe\" (UID: \"897afb16-d413-471a-9a63-12a7503736fe\") "
Sep 18 16:13:43.464867 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.464845    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9198458-9a72-48e5-8e65-0f1eeff11c5a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f9198458-9a72-48e5-8e65-0f1eeff11c5a" (UID: "f9198458-9a72-48e5-8e65-0f1eeff11c5a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:13:43.466407 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.466381    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9198458-9a72-48e5-8e65-0f1eeff11c5a-kube-api-access-qgxsx" (OuterVolumeSpecName: "kube-api-access-qgxsx") pod "f9198458-9a72-48e5-8e65-0f1eeff11c5a" (UID: "f9198458-9a72-48e5-8e65-0f1eeff11c5a"). InnerVolumeSpecName "kube-api-access-qgxsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:13:43.466506 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.466408    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897afb16-d413-471a-9a63-12a7503736fe-kube-api-access-sbfjs" (OuterVolumeSpecName: "kube-api-access-sbfjs") pod "897afb16-d413-471a-9a63-12a7503736fe" (UID: "897afb16-d413-471a-9a63-12a7503736fe"). InnerVolumeSpecName "kube-api-access-sbfjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:13:43.519435 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.519383    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:13:43.565901 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.565850    2546 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9198458-9a72-48e5-8e65-0f1eeff11c5a-extensions-socket-volume\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:13:43.565901 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.565873    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbfjs\" (UniqueName: \"kubernetes.io/projected/897afb16-d413-471a-9a63-12a7503736fe-kube-api-access-sbfjs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:13:43.565901 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.565884    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgxsx\" (UniqueName: \"kubernetes.io/projected/f9198458-9a72-48e5-8e65-0f1eeff11c5a-kube-api-access-qgxsx\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:13:43.660227 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.660197    2546 generic.go:356] "Generic (PLEG): container finished" podID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" containerID="d25ee01038cfc8807cb713611eda06508d566d142473104d7a18c8e94f1eb284" exitCode=0
Sep 18 16:13:43.660351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.660247    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:43.660351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.660286    2546 scope.go:117] "RemoveContainer" containerID="d25ee01038cfc8807cb713611eda06508d566d142473104d7a18c8e94f1eb284"
Sep 18 16:13:43.661772 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.661742    2546 generic.go:356] "Generic (PLEG): container finished" podID="897afb16-d413-471a-9a63-12a7503736fe" containerID="d1fb58ebf3de3878be2759728cd9879cbc21c3836c4ead7f3208c5d53abd8bcf" exitCode=0
Sep 18 16:13:43.661890 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.661786    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:43.661999 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.661977    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:43.662707 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.662678    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:43.664746 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.664725    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5wwns\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="897afb16-d413-471a-9a63-12a7503736fe" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:43.666534 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.666505    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:43.668286 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.668261    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5wwns\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="897afb16-d413-471a-9a63-12a7503736fe" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:43.669212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.669196    2546 scope.go:117] "RemoveContainer" containerID="d25ee01038cfc8807cb713611eda06508d566d142473104d7a18c8e94f1eb284"
Sep 18 16:13:43.669499 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.669476    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25ee01038cfc8807cb713611eda06508d566d142473104d7a18c8e94f1eb284\": container with ID starting with d25ee01038cfc8807cb713611eda06508d566d142473104d7a18c8e94f1eb284 not found: ID does not exist" containerID="d25ee01038cfc8807cb713611eda06508d566d142473104d7a18c8e94f1eb284"
Sep 18 16:13:43.669559 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.669512    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25ee01038cfc8807cb713611eda06508d566d142473104d7a18c8e94f1eb284"} err="failed to get container status \"d25ee01038cfc8807cb713611eda06508d566d142473104d7a18c8e94f1eb284\": rpc error: code = NotFound desc = could not find container \"d25ee01038cfc8807cb713611eda06508d566d142473104d7a18c8e94f1eb284\": container with ID starting with d25ee01038cfc8807cb713611eda06508d566d142473104d7a18c8e94f1eb284 not found: ID does not exist"
Sep 18 16:13:43.669559 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.669536    2546 scope.go:117] "RemoveContainer" containerID="d1fb58ebf3de3878be2759728cd9879cbc21c3836c4ead7f3208c5d53abd8bcf"
Sep 18 16:13:43.670189 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.670123    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-kffhc\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="fa5bac16-0d0d-418b-9866-223a843f69a2" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:43.675865 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.675844    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:43.676909 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.676891    2546 scope.go:117] "RemoveContainer" containerID="d1fb58ebf3de3878be2759728cd9879cbc21c3836c4ead7f3208c5d53abd8bcf"
Sep 18 16:13:43.677218 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.677199    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1fb58ebf3de3878be2759728cd9879cbc21c3836c4ead7f3208c5d53abd8bcf\": container with ID starting with d1fb58ebf3de3878be2759728cd9879cbc21c3836c4ead7f3208c5d53abd8bcf not found: ID does not exist" containerID="d1fb58ebf3de3878be2759728cd9879cbc21c3836c4ead7f3208c5d53abd8bcf"
Sep 18 16:13:43.677276 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.677227    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1fb58ebf3de3878be2759728cd9879cbc21c3836c4ead7f3208c5d53abd8bcf"} err="failed to get container status \"d1fb58ebf3de3878be2759728cd9879cbc21c3836c4ead7f3208c5d53abd8bcf\": rpc error: code = NotFound desc = could not find container \"d1fb58ebf3de3878be2759728cd9879cbc21c3836c4ead7f3208c5d53abd8bcf\": container with ID starting with d1fb58ebf3de3878be2759728cd9879cbc21c3836c4ead7f3208c5d53abd8bcf not found: ID does not exist"
Sep 18 16:13:43.677710 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.677687    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5wwns\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="897afb16-d413-471a-9a63-12a7503736fe" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:43.679681 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.679659    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-kffhc\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="fa5bac16-0d0d-418b-9866-223a843f69a2" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:43.681470 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.681452    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5wwns\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="897afb16-d413-471a-9a63-12a7503736fe" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:43.683129 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.683109    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-kffhc\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="fa5bac16-0d0d-418b-9866-223a843f69a2" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:43.684923 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.684899    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:43.695075 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.695057    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:43.697172 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.697153    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:43.698959 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.698940    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5wwns\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="897afb16-d413-471a-9a63-12a7503736fe" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:43.700925 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:43.700908    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-kffhc\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="fa5bac16-0d0d-418b-9866-223a843f69a2" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:43.729727 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.729705    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"]
Sep 18 16:13:43.731572 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:13:43.731547    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db8b0ba_8ed6_4015_8b68_3b133143fdc3.slice/crio-06bc6b1942ba873e3de7a0e97dd73b9a5672d66cd58f98f524fe461a1951a746 WatchSource:0}: Error finding container 06bc6b1942ba873e3de7a0e97dd73b9a5672d66cd58f98f524fe461a1951a746: Status 404 returned error can't find the container with id 06bc6b1942ba873e3de7a0e97dd73b9a5672d66cd58f98f524fe461a1951a746
Sep 18 16:13:43.759008 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.758985    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"]
Sep 18 16:13:43.760454 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:13:43.760434    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c53eeff_c125_49c6_ab26_182dd935f398.slice/crio-ee13b876ea57adbf3d8e685360b70ffd8861eaebd5822d42ee419feb6edc5614 WatchSource:0}: Error finding container ee13b876ea57adbf3d8e685360b70ffd8861eaebd5822d42ee419feb6edc5614: Status 404 returned error can't find the container with id ee13b876ea57adbf3d8e685360b70ffd8861eaebd5822d42ee419feb6edc5614
Sep 18 16:13:43.766515 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.766496    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa5bac16-0d0d-418b-9866-223a843f69a2-extensions-socket-volume\") pod \"fa5bac16-0d0d-418b-9866-223a843f69a2\" (UID: \"fa5bac16-0d0d-418b-9866-223a843f69a2\") "
Sep 18 16:13:43.766709 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.766695    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvvcs\" (UniqueName: \"kubernetes.io/projected/fa5bac16-0d0d-418b-9866-223a843f69a2-kube-api-access-lvvcs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:13:43.766768 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.766694    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa5bac16-0d0d-418b-9866-223a843f69a2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fa5bac16-0d0d-418b-9866-223a843f69a2" (UID: "fa5bac16-0d0d-418b-9866-223a843f69a2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:13:43.824098 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.824074    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"]
Sep 18 16:13:43.826758 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:13:43.826737    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d67412_30af_4fb0_adc0_5251f88b7a41.slice/crio-b0b9a07a9ca88ed11df2f5dad31d999b1b9b83c3d8fa71383dac59a497f103fe WatchSource:0}: Error finding container b0b9a07a9ca88ed11df2f5dad31d999b1b9b83c3d8fa71383dac59a497f103fe: Status 404 returned error can't find the container with id b0b9a07a9ca88ed11df2f5dad31d999b1b9b83c3d8fa71383dac59a497f103fe
Sep 18 16:13:43.867730 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:43.867704    2546 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fa5bac16-0d0d-418b-9866-223a843f69a2-extensions-socket-volume\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:13:44.600237 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:44.600203    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-5wwns\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="897afb16-d413-471a-9a63-12a7503736fe" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-5wwns"
Sep 18 16:13:44.602470 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.602450    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="897afb16-d413-471a-9a63-12a7503736fe" path="/var/lib/kubelet/pods/897afb16-d413-471a-9a63-12a7503736fe/volumes"
Sep 18 16:13:44.602809 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.602796    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" path="/var/lib/kubelet/pods/f9198458-9a72-48e5-8e65-0f1eeff11c5a/volumes"
Sep 18 16:13:44.603125 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.603108    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa5bac16-0d0d-418b-9866-223a843f69a2" path="/var/lib/kubelet/pods/fa5bac16-0d0d-418b-9866-223a843f69a2/volumes"
Sep 18 16:13:44.603215 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:44.603130    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-kffhc\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="fa5bac16-0d0d-418b-9866-223a843f69a2" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:44.605177 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:44.605135    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-t4msp\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="f9198458-9a72-48e5-8e65-0f1eeff11c5a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-t4msp"
Sep 18 16:13:44.667312 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.667285    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" event={"ID":"1c53eeff-c125-49c6-ab26-182dd935f398","Type":"ContainerStarted","Data":"d7b66f6dcf15f0ee940b6003c3a51567213d18789e0c372e06cdd646e21ee788"}
Sep 18 16:13:44.667443 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.667333    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" event={"ID":"1c53eeff-c125-49c6-ab26-182dd935f398","Type":"ContainerStarted","Data":"ee13b876ea57adbf3d8e685360b70ffd8861eaebd5822d42ee419feb6edc5614"}
Sep 18 16:13:44.667443 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.667359    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:13:44.668827 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.668807    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" event={"ID":"61d67412-30af-4fb0-adc0-5251f88b7a41","Type":"ContainerStarted","Data":"324122dd9d31d30f4c518b31ef077ad5febc35257d54ba486e28086f31a52233"}
Sep 18 16:13:44.668944 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.668854    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:13:44.668944 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.668879    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" event={"ID":"61d67412-30af-4fb0-adc0-5251f88b7a41","Type":"ContainerStarted","Data":"b0b9a07a9ca88ed11df2f5dad31d999b1b9b83c3d8fa71383dac59a497f103fe"}
Sep 18 16:13:44.674445 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.674420    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq" event={"ID":"8db8b0ba-8ed6-4015-8b68-3b133143fdc3","Type":"ContainerStarted","Data":"773ab6f266aab50526b77e553ecd687d9a22002acf53dfb1bb51db946a047c7b"}
Sep 18 16:13:44.674540 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.674461    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"
Sep 18 16:13:44.674540 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.674481    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq" event={"ID":"8db8b0ba-8ed6-4015-8b68-3b133143fdc3","Type":"ContainerStarted","Data":"06bc6b1942ba873e3de7a0e97dd73b9a5672d66cd58f98f524fe461a1951a746"}
Sep 18 16:13:44.675400 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.675385    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:44.693742 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:44.693702    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-kffhc\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="fa5bac16-0d0d-418b-9866-223a843f69a2" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-kffhc"
Sep 18 16:13:44.693955 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.693917    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podStartSLOduration=1.693904396 podStartE2EDuration="1.693904396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:13:44.691449876 +0000 UTC m=+520.600230274" watchObservedRunningTime="2026-09-18 16:13:44.693904396 +0000 UTC m=+520.602684748"
Sep 18 16:13:44.716295 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.716257    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" podStartSLOduration=1.716245199 podStartE2EDuration="1.716245199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:13:44.71509844 +0000 UTC m=+520.623878830" watchObservedRunningTime="2026-09-18 16:13:44.716245199 +0000 UTC m=+520.625025551"
Sep 18 16:13:44.736335 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:44.736301    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq" podStartSLOduration=1.736285515 podStartE2EDuration="1.736285515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:13:44.735302069 +0000 UTC m=+520.644082422" watchObservedRunningTime="2026-09-18 16:13:44.736285515 +0000 UTC m=+520.645065870"
Sep 18 16:13:55.683103 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:55.683074    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"
Sep 18 16:13:55.683691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:55.683147    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:13:55.683691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:55.683193    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:13:55.863163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:55.863131    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"]
Sep 18 16:13:55.863351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:55.863329    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq" podUID="8db8b0ba-8ed6-4015-8b68-3b133143fdc3" containerName="manager" containerID="cri-o://773ab6f266aab50526b77e553ecd687d9a22002acf53dfb1bb51db946a047c7b" gracePeriod=10
Sep 18 16:13:56.112051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.112013    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"
Sep 18 16:13:56.270306 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.270242    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8db8b0ba-8ed6-4015-8b68-3b133143fdc3-extensions-socket-volume\") pod \"8db8b0ba-8ed6-4015-8b68-3b133143fdc3\" (UID: \"8db8b0ba-8ed6-4015-8b68-3b133143fdc3\") "
Sep 18 16:13:56.270427 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.270303    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-779gz\" (UniqueName: \"kubernetes.io/projected/8db8b0ba-8ed6-4015-8b68-3b133143fdc3-kube-api-access-779gz\") pod \"8db8b0ba-8ed6-4015-8b68-3b133143fdc3\" (UID: \"8db8b0ba-8ed6-4015-8b68-3b133143fdc3\") "
Sep 18 16:13:56.270642 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.270622    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db8b0ba-8ed6-4015-8b68-3b133143fdc3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8db8b0ba-8ed6-4015-8b68-3b133143fdc3" (UID: "8db8b0ba-8ed6-4015-8b68-3b133143fdc3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:13:56.272198 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.272179    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db8b0ba-8ed6-4015-8b68-3b133143fdc3-kube-api-access-779gz" (OuterVolumeSpecName: "kube-api-access-779gz") pod "8db8b0ba-8ed6-4015-8b68-3b133143fdc3" (UID: "8db8b0ba-8ed6-4015-8b68-3b133143fdc3"). InnerVolumeSpecName "kube-api-access-779gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:13:56.371862 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.371838    2546 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8db8b0ba-8ed6-4015-8b68-3b133143fdc3-extensions-socket-volume\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:13:56.371862 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.371858    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-779gz\" (UniqueName: \"kubernetes.io/projected/8db8b0ba-8ed6-4015-8b68-3b133143fdc3-kube-api-access-779gz\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:13:56.730823 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.730793    2546 generic.go:356] "Generic (PLEG): container finished" podID="8db8b0ba-8ed6-4015-8b68-3b133143fdc3" containerID="773ab6f266aab50526b77e553ecd687d9a22002acf53dfb1bb51db946a047c7b" exitCode=0
Sep 18 16:13:56.731169 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.730833    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"
Sep 18 16:13:56.731169 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.730878    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq" event={"ID":"8db8b0ba-8ed6-4015-8b68-3b133143fdc3","Type":"ContainerDied","Data":"773ab6f266aab50526b77e553ecd687d9a22002acf53dfb1bb51db946a047c7b"}
Sep 18 16:13:56.731169 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.730925    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq" event={"ID":"8db8b0ba-8ed6-4015-8b68-3b133143fdc3","Type":"ContainerDied","Data":"06bc6b1942ba873e3de7a0e97dd73b9a5672d66cd58f98f524fe461a1951a746"}
Sep 18 16:13:56.731169 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.730947    2546 scope.go:117] "RemoveContainer" containerID="773ab6f266aab50526b77e553ecd687d9a22002acf53dfb1bb51db946a047c7b"
Sep 18 16:13:56.739736 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.739715    2546 scope.go:117] "RemoveContainer" containerID="773ab6f266aab50526b77e553ecd687d9a22002acf53dfb1bb51db946a047c7b"
Sep 18 16:13:56.740148 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:13:56.740009    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773ab6f266aab50526b77e553ecd687d9a22002acf53dfb1bb51db946a047c7b\": container with ID starting with 773ab6f266aab50526b77e553ecd687d9a22002acf53dfb1bb51db946a047c7b not found: ID does not exist" containerID="773ab6f266aab50526b77e553ecd687d9a22002acf53dfb1bb51db946a047c7b"
Sep 18 16:13:56.740218 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.740162    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773ab6f266aab50526b77e553ecd687d9a22002acf53dfb1bb51db946a047c7b"} err="failed to get container status \"773ab6f266aab50526b77e553ecd687d9a22002acf53dfb1bb51db946a047c7b\": rpc error: code = NotFound desc = could not find container \"773ab6f266aab50526b77e553ecd687d9a22002acf53dfb1bb51db946a047c7b\": container with ID starting with 773ab6f266aab50526b77e553ecd687d9a22002acf53dfb1bb51db946a047c7b not found: ID does not exist"
Sep 18 16:13:56.765587 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.765563    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"]
Sep 18 16:13:56.779927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:56.779908    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-56jkq"]
Sep 18 16:13:58.602148 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.602117    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8db8b0ba-8ed6-4015-8b68-3b133143fdc3" path="/var/lib/kubelet/pods/8db8b0ba-8ed6-4015-8b68-3b133143fdc3/volumes"
Sep 18 16:13:58.840475 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.840438    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq"]
Sep 18 16:13:58.840825 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.840808    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db8b0ba-8ed6-4015-8b68-3b133143fdc3" containerName="manager"
Sep 18 16:13:58.840908 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.840828    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db8b0ba-8ed6-4015-8b68-3b133143fdc3" containerName="manager"
Sep 18 16:13:58.840908 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.840902    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db8b0ba-8ed6-4015-8b68-3b133143fdc3" containerName="manager"
Sep 18 16:13:58.843818 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.843797    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq"
Sep 18 16:13:58.857977 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.857925    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rdd2r\""
Sep 18 16:13:58.857977 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.857952    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 16:13:58.857977 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.857965    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 16:13:58.857977 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.857975    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 16:13:58.858284 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.858071    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 16:13:58.863738 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.863716    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq"]
Sep 18 16:13:58.994370 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.994347    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dh4s\" (UniqueName: \"kubernetes.io/projected/e7a16312-0c04-4eb3-92f5-5611d08f6025-kube-api-access-9dh4s\") pod \"managed-serviceaccount-addon-agent-dd9d5946b-hmhtq\" (UID: \"e7a16312-0c04-4eb3-92f5-5611d08f6025\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq"
Sep 18 16:13:58.994494 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:58.994381    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7a16312-0c04-4eb3-92f5-5611d08f6025-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dd9d5946b-hmhtq\" (UID: \"e7a16312-0c04-4eb3-92f5-5611d08f6025\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq"
Sep 18 16:13:59.095001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:59.094965    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7a16312-0c04-4eb3-92f5-5611d08f6025-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dd9d5946b-hmhtq\" (UID: \"e7a16312-0c04-4eb3-92f5-5611d08f6025\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq"
Sep 18 16:13:59.095194 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:59.095109    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dh4s\" (UniqueName: \"kubernetes.io/projected/e7a16312-0c04-4eb3-92f5-5611d08f6025-kube-api-access-9dh4s\") pod \"managed-serviceaccount-addon-agent-dd9d5946b-hmhtq\" (UID: \"e7a16312-0c04-4eb3-92f5-5611d08f6025\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq"
Sep 18 16:13:59.097677 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:59.097648    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7a16312-0c04-4eb3-92f5-5611d08f6025-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-dd9d5946b-hmhtq\" (UID: \"e7a16312-0c04-4eb3-92f5-5611d08f6025\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq"
Sep 18 16:13:59.113745 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:59.113681    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dh4s\" (UniqueName: \"kubernetes.io/projected/e7a16312-0c04-4eb3-92f5-5611d08f6025-kube-api-access-9dh4s\") pod \"managed-serviceaccount-addon-agent-dd9d5946b-hmhtq\" (UID: \"e7a16312-0c04-4eb3-92f5-5611d08f6025\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq"
Sep 18 16:13:59.151939 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:59.151921    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq"
Sep 18 16:13:59.441321 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:59.441300    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq"]
Sep 18 16:13:59.442699 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:13:59.442668    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7a16312_0c04_4eb3_92f5_5611d08f6025.slice/crio-f4187a1582710936da4ff6e42279e41f541de0c27ac637bca1b35539b7a52a54 WatchSource:0}: Error finding container f4187a1582710936da4ff6e42279e41f541de0c27ac637bca1b35539b7a52a54: Status 404 returned error can't find the container with id f4187a1582710936da4ff6e42279e41f541de0c27ac637bca1b35539b7a52a54
Sep 18 16:13:59.746525 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:13:59.746488    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq" event={"ID":"e7a16312-0c04-4eb3-92f5-5611d08f6025","Type":"ContainerStarted","Data":"f4187a1582710936da4ff6e42279e41f541de0c27ac637bca1b35539b7a52a54"}
Sep 18 16:14:02.762904 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:02.762868    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq" event={"ID":"e7a16312-0c04-4eb3-92f5-5611d08f6025","Type":"ContainerStarted","Data":"c0aecfa9682ab38b170281c072a263f50d055085fbe22fbf4aa8d652d46c6a8a"}
Sep 18 16:14:02.797214 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:02.797171    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-dd9d5946b-hmhtq" podStartSLOduration=2.365721374 podStartE2EDuration="4.797153993s" totalImagesPullingTime="2.431432619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:14:02.797106703 +0000 UTC m=+538.705887057" watchObservedRunningTime="2026-09-18 16:14:02.797153993 +0000 UTC m=+538.705934345"
Sep 18 16:14:12.941599 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:12.941563    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"]
Sep 18 16:14:12.946230 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:12.946209    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:12.949661 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:12.949642    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bnd84\""
Sep 18 16:14:12.977924 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:12.977887    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"]
Sep 18 16:14:13.013496 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.013461    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.013496 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.013497    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.013717 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.013535    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.013717 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.013589    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a91800-e401-43ef-9427-597e42651fe1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.013717 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.013657    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.013717 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.013676    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.013717 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.013695    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a91800-e401-43ef-9427-597e42651fe1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.013939 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.013773    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkxd9\" (UniqueName: \"kubernetes.io/projected/43a91800-e401-43ef-9427-597e42651fe1-kube-api-access-rkxd9\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.013939 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.013814    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a91800-e401-43ef-9427-597e42651fe1-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.114464 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.114431    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkxd9\" (UniqueName: \"kubernetes.io/projected/43a91800-e401-43ef-9427-597e42651fe1-kube-api-access-rkxd9\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.114464 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.114470    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a91800-e401-43ef-9427-597e42651fe1-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.114723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.114504    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.114723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.114532    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.114723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.114570    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.114723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.114604    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a91800-e401-43ef-9427-597e42651fe1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.114723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.114671    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.114723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.114697    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.114993 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.114837    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a91800-e401-43ef-9427-597e42651fe1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.115064 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.114998    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.115064 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.115007    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.115204 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.115115    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.115204 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.115182    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.115378 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.115346    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a91800-e401-43ef-9427-597e42651fe1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.117063 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.117019    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a91800-e401-43ef-9427-597e42651fe1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.117275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.117257    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a91800-e401-43ef-9427-597e42651fe1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.133555 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.133528    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a91800-e401-43ef-9427-597e42651fe1-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.136238 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.136212    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkxd9\" (UniqueName: \"kubernetes.io/projected/43a91800-e401-43ef-9427-597e42651fe1-kube-api-access-rkxd9\") pod \"maas-default-gateway-openshift-default-5ccb968c66-chbcf\" (UID: \"43a91800-e401-43ef-9427-597e42651fe1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.256840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.256802    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:13.548328 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.548301    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"]
Sep 18 16:14:13.550150 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:14:13.550125    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a91800_e401_43ef_9427_597e42651fe1.slice/crio-5756da7d8ffef186fe35afaf375f929ff289c97ba7f1eb087b4ff6a168d99e3e WatchSource:0}: Error finding container 5756da7d8ffef186fe35afaf375f929ff289c97ba7f1eb087b4ff6a168d99e3e: Status 404 returned error can't find the container with id 5756da7d8ffef186fe35afaf375f929ff289c97ba7f1eb087b4ff6a168d99e3e
Sep 18 16:14:13.552178 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.552147    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:14:13.552258 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.552217    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:14:13.552258 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.552248    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:14:13.815333 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.815242    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf" event={"ID":"43a91800-e401-43ef-9427-597e42651fe1","Type":"ContainerStarted","Data":"f3dbd0f61daad0328f0888302b4a618d53c5d7b068b06f2a7ff9f167c6a5528f"}
Sep 18 16:14:13.815333 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.815297    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf" event={"ID":"43a91800-e401-43ef-9427-597e42651fe1","Type":"ContainerStarted","Data":"5756da7d8ffef186fe35afaf375f929ff289c97ba7f1eb087b4ff6a168d99e3e"}
Sep 18 16:14:13.858183 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:13.858136    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf" podStartSLOduration=1.858119915 podStartE2EDuration="1.858119915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:14:13.857775981 +0000 UTC m=+549.766556354" watchObservedRunningTime="2026-09-18 16:14:13.858119915 +0000 UTC m=+549.766900267"
Sep 18 16:14:14.257002 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:14.256962    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:14.261790 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:14.261764    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:14.820033 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:14.820002    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:14.821007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:14.820972    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:14:15.788009 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:15.787972    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:14:15.792414 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:15.792399    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"
Sep 18 16:14:15.801690 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:15.801672    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dc6ds\""
Sep 18 16:14:15.801941 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:15.801927    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 16:14:15.808700 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:15.808673    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:14:15.838803 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:15.838772    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7c162150-470f-4f52-bf7a-9cb35c68bb9c-config-file\") pod \"limitador-limitador-84994f7ffc-jw9lb\" (UID: \"7c162150-470f-4f52-bf7a-9cb35c68bb9c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"
Sep 18 16:14:15.838947 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:15.838829    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtmk\" (UniqueName: \"kubernetes.io/projected/7c162150-470f-4f52-bf7a-9cb35c68bb9c-kube-api-access-zvtmk\") pod \"limitador-limitador-84994f7ffc-jw9lb\" (UID: \"7c162150-470f-4f52-bf7a-9cb35c68bb9c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"
Sep 18 16:14:15.857194 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:15.857158    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:14:15.939943 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:15.939904    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7c162150-470f-4f52-bf7a-9cb35c68bb9c-config-file\") pod \"limitador-limitador-84994f7ffc-jw9lb\" (UID: \"7c162150-470f-4f52-bf7a-9cb35c68bb9c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"
Sep 18 16:14:15.940149 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:15.939965    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtmk\" (UniqueName: \"kubernetes.io/projected/7c162150-470f-4f52-bf7a-9cb35c68bb9c-kube-api-access-zvtmk\") pod \"limitador-limitador-84994f7ffc-jw9lb\" (UID: \"7c162150-470f-4f52-bf7a-9cb35c68bb9c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"
Sep 18 16:14:15.940517 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:15.940499    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7c162150-470f-4f52-bf7a-9cb35c68bb9c-config-file\") pod \"limitador-limitador-84994f7ffc-jw9lb\" (UID: \"7c162150-470f-4f52-bf7a-9cb35c68bb9c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"
Sep 18 16:14:15.954164 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:15.954136    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtmk\" (UniqueName: \"kubernetes.io/projected/7c162150-470f-4f52-bf7a-9cb35c68bb9c-kube-api-access-zvtmk\") pod \"limitador-limitador-84994f7ffc-jw9lb\" (UID: \"7c162150-470f-4f52-bf7a-9cb35c68bb9c\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"
Sep 18 16:14:16.102634 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:16.102554    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"
Sep 18 16:14:16.216692 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:16.216651    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vt9nh"]
Sep 18 16:14:16.221042 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:16.220999    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vt9nh"
Sep 18 16:14:16.224853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:16.224837    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vd96t\""
Sep 18 16:14:16.242890 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:16.242860    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbrh5\" (UniqueName: \"kubernetes.io/projected/70ca77d6-409d-4406-9b37-4dd4d6cb1a00-kube-api-access-dbrh5\") pod \"authorino-7bcfd8f9cd-vt9nh\" (UID: \"70ca77d6-409d-4406-9b37-4dd4d6cb1a00\") " pod="kuadrant-system/authorino-7bcfd8f9cd-vt9nh"
Sep 18 16:14:16.243262 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:16.243243    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vt9nh"]
Sep 18 16:14:16.344132 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:16.344099    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbrh5\" (UniqueName: \"kubernetes.io/projected/70ca77d6-409d-4406-9b37-4dd4d6cb1a00-kube-api-access-dbrh5\") pod \"authorino-7bcfd8f9cd-vt9nh\" (UID: \"70ca77d6-409d-4406-9b37-4dd4d6cb1a00\") " pod="kuadrant-system/authorino-7bcfd8f9cd-vt9nh"
Sep 18 16:14:16.358325 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:16.358265    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbrh5\" (UniqueName: \"kubernetes.io/projected/70ca77d6-409d-4406-9b37-4dd4d6cb1a00-kube-api-access-dbrh5\") pod \"authorino-7bcfd8f9cd-vt9nh\" (UID: \"70ca77d6-409d-4406-9b37-4dd4d6cb1a00\") " pod="kuadrant-system/authorino-7bcfd8f9cd-vt9nh"
Sep 18 16:14:16.382609 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:16.382573    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:14:16.384748 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:14:16.384723    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c162150_470f_4f52_bf7a_9cb35c68bb9c.slice/crio-8c82e0121fb5e3f0b5fab0acab5d154f51614dedfb8fe4bf33d0987d5d9fd871 WatchSource:0}: Error finding container 8c82e0121fb5e3f0b5fab0acab5d154f51614dedfb8fe4bf33d0987d5d9fd871: Status 404 returned error can't find the container with id 8c82e0121fb5e3f0b5fab0acab5d154f51614dedfb8fe4bf33d0987d5d9fd871
Sep 18 16:14:16.531222 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:16.531187    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vt9nh"
Sep 18 16:14:16.817937 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:16.817915    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vt9nh"]
Sep 18 16:14:16.820311 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:14:16.820281    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70ca77d6_409d_4406_9b37_4dd4d6cb1a00.slice/crio-a4957adf01a843ef72b519f602157d8b09446b319ed9591dc566c04a10b356b1 WatchSource:0}: Error finding container a4957adf01a843ef72b519f602157d8b09446b319ed9591dc566c04a10b356b1: Status 404 returned error can't find the container with id a4957adf01a843ef72b519f602157d8b09446b319ed9591dc566c04a10b356b1
Sep 18 16:14:16.830638 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:16.830609    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vt9nh" event={"ID":"70ca77d6-409d-4406-9b37-4dd4d6cb1a00","Type":"ContainerStarted","Data":"a4957adf01a843ef72b519f602157d8b09446b319ed9591dc566c04a10b356b1"}
Sep 18 16:14:16.831937 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:16.831912    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw9lb" event={"ID":"7c162150-470f-4f52-bf7a-9cb35c68bb9c","Type":"ContainerStarted","Data":"8c82e0121fb5e3f0b5fab0acab5d154f51614dedfb8fe4bf33d0987d5d9fd871"}
Sep 18 16:14:18.844217 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:18.844174    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vt9nh" event={"ID":"70ca77d6-409d-4406-9b37-4dd4d6cb1a00","Type":"ContainerStarted","Data":"5f987a119a182a6c69f5c6c8374161d461cb70408dd31e88b949296f05a33cd5"}
Sep 18 16:14:18.845768 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:18.845739    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw9lb" event={"ID":"7c162150-470f-4f52-bf7a-9cb35c68bb9c","Type":"ContainerStarted","Data":"1cb5143c421c1b8ebb56abc83cf5e01c051473af3b58ae9e927d49e719a07b7e"}
Sep 18 16:14:18.845883 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:18.845828    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"
Sep 18 16:14:18.896202 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:18.895946    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw9lb" podStartSLOduration=1.860576745 podStartE2EDuration="3.895924242s" totalImagesPullingTime="2.035347497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:14:18.895423685 +0000 UTC m=+554.804204037" watchObservedRunningTime="2026-09-18 16:14:18.895924242 +0000 UTC m=+554.804704597"
Sep 18 16:14:18.896897 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:18.896863    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-vt9nh" podStartSLOduration=1.292768849 podStartE2EDuration="2.896851564s" totalImagesPullingTime="1.604082715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:14:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:14:18.869273309 +0000 UTC m=+554.778053661" watchObservedRunningTime="2026-09-18 16:14:18.896851564 +0000 UTC m=+554.805631917"
Sep 18 16:14:19.690180 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:19.690151    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vt9nh"]
Sep 18 16:14:20.854816 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:20.854780    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-vt9nh" podUID="70ca77d6-409d-4406-9b37-4dd4d6cb1a00" containerName="authorino" containerID="cri-o://5f987a119a182a6c69f5c6c8374161d461cb70408dd31e88b949296f05a33cd5" gracePeriod=30
Sep 18 16:14:21.093730 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:21.093705    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vt9nh"
Sep 18 16:14:21.186523 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:21.186459    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbrh5\" (UniqueName: \"kubernetes.io/projected/70ca77d6-409d-4406-9b37-4dd4d6cb1a00-kube-api-access-dbrh5\") pod \"70ca77d6-409d-4406-9b37-4dd4d6cb1a00\" (UID: \"70ca77d6-409d-4406-9b37-4dd4d6cb1a00\") "
Sep 18 16:14:21.188467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:21.188446    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ca77d6-409d-4406-9b37-4dd4d6cb1a00-kube-api-access-dbrh5" (OuterVolumeSpecName: "kube-api-access-dbrh5") pod "70ca77d6-409d-4406-9b37-4dd4d6cb1a00" (UID: "70ca77d6-409d-4406-9b37-4dd4d6cb1a00"). InnerVolumeSpecName "kube-api-access-dbrh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:14:21.287831 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:21.287802    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbrh5\" (UniqueName: \"kubernetes.io/projected/70ca77d6-409d-4406-9b37-4dd4d6cb1a00-kube-api-access-dbrh5\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:14:21.860608 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:21.860571    2546 generic.go:356] "Generic (PLEG): container finished" podID="70ca77d6-409d-4406-9b37-4dd4d6cb1a00" containerID="5f987a119a182a6c69f5c6c8374161d461cb70408dd31e88b949296f05a33cd5" exitCode=0
Sep 18 16:14:21.861047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:21.860615    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vt9nh"
Sep 18 16:14:21.861047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:21.860656    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vt9nh" event={"ID":"70ca77d6-409d-4406-9b37-4dd4d6cb1a00","Type":"ContainerDied","Data":"5f987a119a182a6c69f5c6c8374161d461cb70408dd31e88b949296f05a33cd5"}
Sep 18 16:14:21.861047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:21.860701    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vt9nh" event={"ID":"70ca77d6-409d-4406-9b37-4dd4d6cb1a00","Type":"ContainerDied","Data":"a4957adf01a843ef72b519f602157d8b09446b319ed9591dc566c04a10b356b1"}
Sep 18 16:14:21.861047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:21.860721    2546 scope.go:117] "RemoveContainer" containerID="5f987a119a182a6c69f5c6c8374161d461cb70408dd31e88b949296f05a33cd5"
Sep 18 16:14:21.868963 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:21.868939    2546 scope.go:117] "RemoveContainer" containerID="5f987a119a182a6c69f5c6c8374161d461cb70408dd31e88b949296f05a33cd5"
Sep 18 16:14:21.869238 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:14:21.869222    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f987a119a182a6c69f5c6c8374161d461cb70408dd31e88b949296f05a33cd5\": container with ID starting with 5f987a119a182a6c69f5c6c8374161d461cb70408dd31e88b949296f05a33cd5 not found: ID does not exist" containerID="5f987a119a182a6c69f5c6c8374161d461cb70408dd31e88b949296f05a33cd5"
Sep 18 16:14:21.869288 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:21.869246    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f987a119a182a6c69f5c6c8374161d461cb70408dd31e88b949296f05a33cd5"} err="failed to get container status \"5f987a119a182a6c69f5c6c8374161d461cb70408dd31e88b949296f05a33cd5\": rpc error: code = NotFound desc = could not find container \"5f987a119a182a6c69f5c6c8374161d461cb70408dd31e88b949296f05a33cd5\": container with ID starting with 5f987a119a182a6c69f5c6c8374161d461cb70408dd31e88b949296f05a33cd5 not found: ID does not exist"
Sep 18 16:14:21.886754 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:21.886732    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vt9nh"]
Sep 18 16:14:21.888265 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:21.888247    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vt9nh"]
Sep 18 16:14:22.602373 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:22.602342    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70ca77d6-409d-4406-9b37-4dd4d6cb1a00" path="/var/lib/kubelet/pods/70ca77d6-409d-4406-9b37-4dd4d6cb1a00/volumes"
Sep 18 16:14:29.851741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:29.851708    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"
Sep 18 16:14:44.372967 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.372928    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-blcf6"]
Sep 18 16:14:44.373448 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.373432    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ca77d6-409d-4406-9b37-4dd4d6cb1a00" containerName="authorino"
Sep 18 16:14:44.373493 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.373451    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ca77d6-409d-4406-9b37-4dd4d6cb1a00" containerName="authorino"
Sep 18 16:14:44.373542 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.373531    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ca77d6-409d-4406-9b37-4dd4d6cb1a00" containerName="authorino"
Sep 18 16:14:44.380838 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.380816    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-blcf6"
Sep 18 16:14:44.386880 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.386861    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vd96t\""
Sep 18 16:14:44.391692 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.391670    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-blcf6"]
Sep 18 16:14:44.481887 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.481853    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d498\" (UniqueName: \"kubernetes.io/projected/cebf4ce6-5343-49f1-bb71-aa27c5257e35-kube-api-access-8d498\") pod \"authorino-557f876f69-blcf6\" (UID: \"cebf4ce6-5343-49f1-bb71-aa27c5257e35\") " pod="kuadrant-system/authorino-557f876f69-blcf6"
Sep 18 16:14:44.582738 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.582704    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d498\" (UniqueName: \"kubernetes.io/projected/cebf4ce6-5343-49f1-bb71-aa27c5257e35-kube-api-access-8d498\") pod \"authorino-557f876f69-blcf6\" (UID: \"cebf4ce6-5343-49f1-bb71-aa27c5257e35\") " pod="kuadrant-system/authorino-557f876f69-blcf6"
Sep 18 16:14:44.594947 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.594918    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d498\" (UniqueName: \"kubernetes.io/projected/cebf4ce6-5343-49f1-bb71-aa27c5257e35-kube-api-access-8d498\") pod \"authorino-557f876f69-blcf6\" (UID: \"cebf4ce6-5343-49f1-bb71-aa27c5257e35\") " pod="kuadrant-system/authorino-557f876f69-blcf6"
Sep 18 16:14:44.646958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.646880    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-blcf6"]
Sep 18 16:14:44.647187 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.647171    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-blcf6"
Sep 18 16:14:44.697777 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.697563    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-785d9fd5bd-4f786"]
Sep 18 16:14:44.702586 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.702564    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-785d9fd5bd-4f786"
Sep 18 16:14:44.713850 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.713753    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-785d9fd5bd-4f786"]
Sep 18 16:14:44.784553 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.784522    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gklm6\" (UniqueName: \"kubernetes.io/projected/f8238aa8-8393-4f32-a1a6-6778ed089f4e-kube-api-access-gklm6\") pod \"authorino-785d9fd5bd-4f786\" (UID: \"f8238aa8-8393-4f32-a1a6-6778ed089f4e\") " pod="kuadrant-system/authorino-785d9fd5bd-4f786"
Sep 18 16:14:44.819739 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.819715    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-785d9fd5bd-4f786"]
Sep 18 16:14:44.819953 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:14:44.819937    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gklm6], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-785d9fd5bd-4f786" podUID="f8238aa8-8393-4f32-a1a6-6778ed089f4e"
Sep 18 16:14:44.884285 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.884256    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79569cf898-5sd2p"]
Sep 18 16:14:44.885058 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.885009    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklm6\" (UniqueName: \"kubernetes.io/projected/f8238aa8-8393-4f32-a1a6-6778ed089f4e-kube-api-access-gklm6\") pod \"authorino-785d9fd5bd-4f786\" (UID: \"f8238aa8-8393-4f32-a1a6-6778ed089f4e\") " pod="kuadrant-system/authorino-785d9fd5bd-4f786"
Sep 18 16:14:44.887739 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.887719    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79569cf898-5sd2p"
Sep 18 16:14:44.892785 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.892767    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 16:14:44.903898 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.903848    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79569cf898-5sd2p"]
Sep 18 16:14:44.905219 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.905196    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gklm6\" (UniqueName: \"kubernetes.io/projected/f8238aa8-8393-4f32-a1a6-6778ed089f4e-kube-api-access-gklm6\") pod \"authorino-785d9fd5bd-4f786\" (UID: \"f8238aa8-8393-4f32-a1a6-6778ed089f4e\") " pod="kuadrant-system/authorino-785d9fd5bd-4f786"
Sep 18 16:14:44.934606 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.934584    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-blcf6"]
Sep 18 16:14:44.936436 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:14:44.936411    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcebf4ce6_5343_49f1_bb71_aa27c5257e35.slice/crio-aece5b142567c62a36eb52366798712746721416e3ffdc11bdf5633bcc4d2abc WatchSource:0}: Error finding container aece5b142567c62a36eb52366798712746721416e3ffdc11bdf5633bcc4d2abc: Status 404 returned error can't find the container with id aece5b142567c62a36eb52366798712746721416e3ffdc11bdf5633bcc4d2abc
Sep 18 16:14:44.970072 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.970054    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-785d9fd5bd-4f786"
Sep 18 16:14:44.970164 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.970049    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-blcf6" event={"ID":"cebf4ce6-5343-49f1-bb71-aa27c5257e35","Type":"ContainerStarted","Data":"aece5b142567c62a36eb52366798712746721416e3ffdc11bdf5633bcc4d2abc"}
Sep 18 16:14:44.977012 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.976994    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-785d9fd5bd-4f786"
Sep 18 16:14:44.985907 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.985890    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22dc720e-bf48-490e-883c-fcb3225cb55f-tls-cert\") pod \"authorino-79569cf898-5sd2p\" (UID: \"22dc720e-bf48-490e-883c-fcb3225cb55f\") " pod="kuadrant-system/authorino-79569cf898-5sd2p"
Sep 18 16:14:44.986004 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:44.985935    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nlm6\" (UniqueName: \"kubernetes.io/projected/22dc720e-bf48-490e-883c-fcb3225cb55f-kube-api-access-9nlm6\") pod \"authorino-79569cf898-5sd2p\" (UID: \"22dc720e-bf48-490e-883c-fcb3225cb55f\") " pod="kuadrant-system/authorino-79569cf898-5sd2p"
Sep 18 16:14:45.086964 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:45.086931    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gklm6\" (UniqueName: \"kubernetes.io/projected/f8238aa8-8393-4f32-a1a6-6778ed089f4e-kube-api-access-gklm6\") pod \"f8238aa8-8393-4f32-a1a6-6778ed089f4e\" (UID: \"f8238aa8-8393-4f32-a1a6-6778ed089f4e\") "
Sep 18 16:14:45.087152 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:45.087137    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22dc720e-bf48-490e-883c-fcb3225cb55f-tls-cert\") pod \"authorino-79569cf898-5sd2p\" (UID: \"22dc720e-bf48-490e-883c-fcb3225cb55f\") " pod="kuadrant-system/authorino-79569cf898-5sd2p"
Sep 18 16:14:45.087227 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:45.087189    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nlm6\" (UniqueName: \"kubernetes.io/projected/22dc720e-bf48-490e-883c-fcb3225cb55f-kube-api-access-9nlm6\") pod \"authorino-79569cf898-5sd2p\" (UID: \"22dc720e-bf48-490e-883c-fcb3225cb55f\") " pod="kuadrant-system/authorino-79569cf898-5sd2p"
Sep 18 16:14:45.089097 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:45.089063    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8238aa8-8393-4f32-a1a6-6778ed089f4e-kube-api-access-gklm6" (OuterVolumeSpecName: "kube-api-access-gklm6") pod "f8238aa8-8393-4f32-a1a6-6778ed089f4e" (UID: "f8238aa8-8393-4f32-a1a6-6778ed089f4e"). InnerVolumeSpecName "kube-api-access-gklm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:14:45.089523 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:45.089504    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22dc720e-bf48-490e-883c-fcb3225cb55f-tls-cert\") pod \"authorino-79569cf898-5sd2p\" (UID: \"22dc720e-bf48-490e-883c-fcb3225cb55f\") " pod="kuadrant-system/authorino-79569cf898-5sd2p"
Sep 18 16:14:45.106583 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:45.106559    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nlm6\" (UniqueName: \"kubernetes.io/projected/22dc720e-bf48-490e-883c-fcb3225cb55f-kube-api-access-9nlm6\") pod \"authorino-79569cf898-5sd2p\" (UID: \"22dc720e-bf48-490e-883c-fcb3225cb55f\") " pod="kuadrant-system/authorino-79569cf898-5sd2p"
Sep 18 16:14:45.188289 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:45.188240    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gklm6\" (UniqueName: \"kubernetes.io/projected/f8238aa8-8393-4f32-a1a6-6778ed089f4e-kube-api-access-gklm6\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:14:45.196142 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:45.196124    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79569cf898-5sd2p"
Sep 18 16:14:45.480261 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:45.480191    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79569cf898-5sd2p"]
Sep 18 16:14:45.482555 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:14:45.482528    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22dc720e_bf48_490e_883c_fcb3225cb55f.slice/crio-57f53b851786e3e6276d9b8843a2cc4ab4e938d7e36447b7878c12016f08513f WatchSource:0}: Error finding container 57f53b851786e3e6276d9b8843a2cc4ab4e938d7e36447b7878c12016f08513f: Status 404 returned error can't find the container with id 57f53b851786e3e6276d9b8843a2cc4ab4e938d7e36447b7878c12016f08513f
Sep 18 16:14:45.977438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:45.977411    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-785d9fd5bd-4f786"
Sep 18 16:14:45.977438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:45.977435    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79569cf898-5sd2p" event={"ID":"22dc720e-bf48-490e-883c-fcb3225cb55f","Type":"ContainerStarted","Data":"07bd3e788e207b563ef964aa3289482d80ce00cdb069bad518e11b8154608dbe"}
Sep 18 16:14:45.977673 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:45.977468    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79569cf898-5sd2p" event={"ID":"22dc720e-bf48-490e-883c-fcb3225cb55f","Type":"ContainerStarted","Data":"57f53b851786e3e6276d9b8843a2cc4ab4e938d7e36447b7878c12016f08513f"}
Sep 18 16:14:46.016193 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:46.016133    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-79569cf898-5sd2p" podStartSLOduration=1.66687198 podStartE2EDuration="2.016114109s" totalImagesPullingTime="349.242129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:14:46.008759559 +0000 UTC m=+581.917539913" watchObservedRunningTime="2026-09-18 16:14:46.016114109 +0000 UTC m=+581.924894461"
Sep 18 16:14:46.092870 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:46.092802    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-785d9fd5bd-4f786"]
Sep 18 16:14:46.101851 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:46.101823    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-785d9fd5bd-4f786"]
Sep 18 16:14:46.602398 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:46.602322    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8238aa8-8393-4f32-a1a6-6778ed089f4e" path="/var/lib/kubelet/pods/f8238aa8-8393-4f32-a1a6-6778ed089f4e/volumes"
Sep 18 16:14:46.983235 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:46.983196    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-blcf6" podUID="cebf4ce6-5343-49f1-bb71-aa27c5257e35" containerName="authorino" containerID="cri-o://817e07d3445969253b5524ae166e984e05f22d51899961f81e5339562f442237" gracePeriod=30
Sep 18 16:14:46.983419 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:46.983200    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-blcf6" event={"ID":"cebf4ce6-5343-49f1-bb71-aa27c5257e35","Type":"ContainerStarted","Data":"817e07d3445969253b5524ae166e984e05f22d51899961f81e5339562f442237"}
Sep 18 16:14:47.016666 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:47.016621    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-blcf6" podStartSLOduration=1.656931494 podStartE2EDuration="3.016606607s" totalImagesPullingTime="1.359675113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:14:47.014620093 +0000 UTC m=+582.923400447" watchObservedRunningTime="2026-09-18 16:14:47.016606607 +0000 UTC m=+582.925386958"
Sep 18 16:14:47.222513 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:47.222487    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-blcf6"
Sep 18 16:14:47.306149 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:47.306079    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d498\" (UniqueName: \"kubernetes.io/projected/cebf4ce6-5343-49f1-bb71-aa27c5257e35-kube-api-access-8d498\") pod \"cebf4ce6-5343-49f1-bb71-aa27c5257e35\" (UID: \"cebf4ce6-5343-49f1-bb71-aa27c5257e35\") "
Sep 18 16:14:47.307992 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:47.307962    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cebf4ce6-5343-49f1-bb71-aa27c5257e35-kube-api-access-8d498" (OuterVolumeSpecName: "kube-api-access-8d498") pod "cebf4ce6-5343-49f1-bb71-aa27c5257e35" (UID: "cebf4ce6-5343-49f1-bb71-aa27c5257e35"). InnerVolumeSpecName "kube-api-access-8d498". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:14:47.407508 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:47.407471    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d498\" (UniqueName: \"kubernetes.io/projected/cebf4ce6-5343-49f1-bb71-aa27c5257e35-kube-api-access-8d498\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:14:47.989417 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:47.989388    2546 generic.go:356] "Generic (PLEG): container finished" podID="cebf4ce6-5343-49f1-bb71-aa27c5257e35" containerID="817e07d3445969253b5524ae166e984e05f22d51899961f81e5339562f442237" exitCode=0
Sep 18 16:14:47.989785 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:47.989427    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-blcf6"
Sep 18 16:14:47.989785 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:47.989475    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-blcf6" event={"ID":"cebf4ce6-5343-49f1-bb71-aa27c5257e35","Type":"ContainerDied","Data":"817e07d3445969253b5524ae166e984e05f22d51899961f81e5339562f442237"}
Sep 18 16:14:47.989785 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:47.989519    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-blcf6" event={"ID":"cebf4ce6-5343-49f1-bb71-aa27c5257e35","Type":"ContainerDied","Data":"aece5b142567c62a36eb52366798712746721416e3ffdc11bdf5633bcc4d2abc"}
Sep 18 16:14:47.989785 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:47.989540    2546 scope.go:117] "RemoveContainer" containerID="817e07d3445969253b5524ae166e984e05f22d51899961f81e5339562f442237"
Sep 18 16:14:47.997695 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:47.997676    2546 scope.go:117] "RemoveContainer" containerID="817e07d3445969253b5524ae166e984e05f22d51899961f81e5339562f442237"
Sep 18 16:14:47.997972 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:14:47.997956    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817e07d3445969253b5524ae166e984e05f22d51899961f81e5339562f442237\": container with ID starting with 817e07d3445969253b5524ae166e984e05f22d51899961f81e5339562f442237 not found: ID does not exist" containerID="817e07d3445969253b5524ae166e984e05f22d51899961f81e5339562f442237"
Sep 18 16:14:47.998056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:47.997978    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817e07d3445969253b5524ae166e984e05f22d51899961f81e5339562f442237"} err="failed to get container status \"817e07d3445969253b5524ae166e984e05f22d51899961f81e5339562f442237\": rpc error: code = NotFound desc = could not find container \"817e07d3445969253b5524ae166e984e05f22d51899961f81e5339562f442237\": container with ID starting with 817e07d3445969253b5524ae166e984e05f22d51899961f81e5339562f442237 not found: ID does not exist"
Sep 18 16:14:48.024880 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:48.024855    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-blcf6"]
Sep 18 16:14:48.034166 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:48.034144    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-blcf6"]
Sep 18 16:14:48.602485 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:14:48.602456    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cebf4ce6-5343-49f1-bb71-aa27c5257e35" path="/var/lib/kubelet/pods/cebf4ce6-5343-49f1-bb71-aa27c5257e35/volumes"
Sep 18 16:15:31.915531 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:31.915449    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:15:40.249162 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:40.249128    2546 generic.go:356] "Generic (PLEG): container finished" podID="43a91800-e401-43ef-9427-597e42651fe1" containerID="f3dbd0f61daad0328f0888302b4a618d53c5d7b068b06f2a7ff9f167c6a5528f" exitCode=0
Sep 18 16:15:40.249558 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:40.249203    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf" event={"ID":"43a91800-e401-43ef-9427-597e42651fe1","Type":"ContainerDied","Data":"f3dbd0f61daad0328f0888302b4a618d53c5d7b068b06f2a7ff9f167c6a5528f"}
Sep 18 16:15:40.249600 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:40.249577    2546 scope.go:117] "RemoveContainer" containerID="f3dbd0f61daad0328f0888302b4a618d53c5d7b068b06f2a7ff9f167c6a5528f"
Sep 18 16:15:40.250661 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:40.250630    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:15:40.250721 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:40.250699    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:15:40.250755 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:40.250734    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:15:41.257015 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:41.256976    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf" event={"ID":"43a91800-e401-43ef-9427-597e42651fe1","Type":"ContainerStarted","Data":"aec134de62286e131cd64bc603f21b48e0fb06a86ff5c752fdf2ab2e1b0977cc"}
Sep 18 16:15:42.257830 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:42.257795    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:15:42.259012 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:42.258975    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-chbcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 18 16:15:42.259175 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:42.259045    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf" podUID="43a91800-e401-43ef-9427-597e42651fe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 18 16:15:43.257602 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:43.257568    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-chbcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 18 16:15:43.257786 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:43.257625    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf" podUID="43a91800-e401-43ef-9427-597e42651fe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 18 16:15:44.257916 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:44.257882    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-chbcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 18 16:15:44.258343 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:44.257938    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf" podUID="43a91800-e401-43ef-9427-597e42651fe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 18 16:15:44.819959 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:44.819905    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:15:45.258018 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:45.257980    2546 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-chbcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 18 16:15:45.258436 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:45.258070    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf" podUID="43a91800-e401-43ef-9427-597e42651fe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 18 16:15:46.261925 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:46.261893    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:15:46.262931 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:46.262911    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-chbcf"
Sep 18 16:15:56.782210 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:15:56.782168    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:16:05.724207 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:16:05.724172    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:16:28.232897 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:16:28.232862    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:16:36.399581 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:16:36.399530    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:17:00.501454 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:00.501373    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:17:08.483455 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:08.483409    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:17:39.539428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.539390    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"]
Sep 18 16:17:39.539902 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.539736    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cebf4ce6-5343-49f1-bb71-aa27c5257e35" containerName="authorino"
Sep 18 16:17:39.539902 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.539747    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cebf4ce6-5343-49f1-bb71-aa27c5257e35" containerName="authorino"
Sep 18 16:17:39.539902 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.539806    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cebf4ce6-5343-49f1-bb71-aa27c5257e35" containerName="authorino"
Sep 18 16:17:39.542718 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.542702    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.559977 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.559956    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 16:17:39.560245 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.560232    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 16:17:39.565119 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.565098    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 18 16:17:39.565228 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.565188    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sr7qq\""
Sep 18 16:17:39.571653 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.571633    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"]
Sep 18 16:17:39.613884 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.613860    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.614049 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.613893    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.614049 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.614015    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.614164 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.614063    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.614164 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.614085    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8mtb\" (UniqueName: \"kubernetes.io/projected/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-kube-api-access-g8mtb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.614164 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.614140    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.714774 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.714734    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.714774 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.714773    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.715060 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.714792    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8mtb\" (UniqueName: \"kubernetes.io/projected/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-kube-api-access-g8mtb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.715060 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.714819    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.715060 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.714838    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.715060 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.714870    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.715274 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.715243    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.715316 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.715293    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.715349 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.715321    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.716988 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.716962    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.717303 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.717281    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.739629 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.739599    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8mtb\" (UniqueName: \"kubernetes.io/projected/b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2-kube-api-access-g8mtb\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z\" (UID: \"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:39.852075 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:39.851984    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:40.145077 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:40.144191    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"]
Sep 18 16:17:40.149291 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:40.149268    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:17:40.836244 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:40.836208    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z" event={"ID":"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2","Type":"ContainerStarted","Data":"fc9ec5f1e2d5858fd77fd6313d326692a19d50d1e0713e391a68c2c529fe09fd"}
Sep 18 16:17:41.842122 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:41.842083    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z" event={"ID":"b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2","Type":"ContainerStarted","Data":"da47c31717ee4e3670c741d7108ade51b982880202c17fc8c286b6ec9b6e221b"}
Sep 18 16:17:41.871244 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:41.871186    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z" podStartSLOduration=1.9113157790000002 podStartE2EDuration="2.871169143s" totalImagesPullingTime="959.853364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:41.86940596 +0000 UTC m=+757.778186321" watchObservedRunningTime="2026-09-18 16:17:41.871169143 +0000 UTC m=+757.779949493"
Sep 18 16:17:42.215822 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:42.215781    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:17:49.852751 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:49.852714    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:49.852751 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:49.852761    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:49.864763 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:49.864740    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:17:49.892252 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:17:49.892226    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z"
Sep 18 16:19:01.415967 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:01.415933    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-766f75f9d7-btp4s"]
Sep 18 16:19:01.419648 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:01.419627    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-766f75f9d7-btp4s"
Sep 18 16:19:01.444425 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:01.444395    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-766f75f9d7-btp4s"]
Sep 18 16:19:01.504206 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:01.504172    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csjtx\" (UniqueName: \"kubernetes.io/projected/fa3e43bc-9fbf-4748-a4a6-b497c0556912-kube-api-access-csjtx\") pod \"authorino-766f75f9d7-btp4s\" (UID: \"fa3e43bc-9fbf-4748-a4a6-b497c0556912\") " pod="kuadrant-system/authorino-766f75f9d7-btp4s"
Sep 18 16:19:01.504362 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:01.504226    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa3e43bc-9fbf-4748-a4a6-b497c0556912-tls-cert\") pod \"authorino-766f75f9d7-btp4s\" (UID: \"fa3e43bc-9fbf-4748-a4a6-b497c0556912\") " pod="kuadrant-system/authorino-766f75f9d7-btp4s"
Sep 18 16:19:01.604795 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:01.604760    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa3e43bc-9fbf-4748-a4a6-b497c0556912-tls-cert\") pod \"authorino-766f75f9d7-btp4s\" (UID: \"fa3e43bc-9fbf-4748-a4a6-b497c0556912\") " pod="kuadrant-system/authorino-766f75f9d7-btp4s"
Sep 18 16:19:01.604960 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:01.604843    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csjtx\" (UniqueName: \"kubernetes.io/projected/fa3e43bc-9fbf-4748-a4a6-b497c0556912-kube-api-access-csjtx\") pod \"authorino-766f75f9d7-btp4s\" (UID: \"fa3e43bc-9fbf-4748-a4a6-b497c0556912\") " pod="kuadrant-system/authorino-766f75f9d7-btp4s"
Sep 18 16:19:01.607410 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:01.607374    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa3e43bc-9fbf-4748-a4a6-b497c0556912-tls-cert\") pod \"authorino-766f75f9d7-btp4s\" (UID: \"fa3e43bc-9fbf-4748-a4a6-b497c0556912\") " pod="kuadrant-system/authorino-766f75f9d7-btp4s"
Sep 18 16:19:01.625205 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:01.625179    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csjtx\" (UniqueName: \"kubernetes.io/projected/fa3e43bc-9fbf-4748-a4a6-b497c0556912-kube-api-access-csjtx\") pod \"authorino-766f75f9d7-btp4s\" (UID: \"fa3e43bc-9fbf-4748-a4a6-b497c0556912\") " pod="kuadrant-system/authorino-766f75f9d7-btp4s"
Sep 18 16:19:01.728995 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:01.728962    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-766f75f9d7-btp4s"
Sep 18 16:19:02.004212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:02.004189    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-766f75f9d7-btp4s"]
Sep 18 16:19:02.006290 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:19:02.006258    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa3e43bc_9fbf_4748_a4a6_b497c0556912.slice/crio-8ab27545d797ddaf5a45effcc4a019374541030a414b6779484a150c51f89594 WatchSource:0}: Error finding container 8ab27545d797ddaf5a45effcc4a019374541030a414b6779484a150c51f89594: Status 404 returned error can't find the container with id 8ab27545d797ddaf5a45effcc4a019374541030a414b6779484a150c51f89594
Sep 18 16:19:02.224525 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:02.224494    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-766f75f9d7-btp4s" event={"ID":"fa3e43bc-9fbf-4748-a4a6-b497c0556912","Type":"ContainerStarted","Data":"8ab27545d797ddaf5a45effcc4a019374541030a414b6779484a150c51f89594"}
Sep 18 16:19:03.233461 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:03.233370    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-766f75f9d7-btp4s" event={"ID":"fa3e43bc-9fbf-4748-a4a6-b497c0556912","Type":"ContainerStarted","Data":"5a3d26e9614c8a2a9d2659c357875886b4d4cdee125d7ffb91c25fee573c80f5"}
Sep 18 16:19:03.259416 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:03.259351    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-766f75f9d7-btp4s" podStartSLOduration=1.32362243 podStartE2EDuration="2.259335522s" totalImagesPullingTime="935.713092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:03.255795049 +0000 UTC m=+839.164575400" watchObservedRunningTime="2026-09-18 16:19:03.259335522 +0000 UTC m=+839.168115870"
Sep 18 16:19:03.318226 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:03.318003    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79569cf898-5sd2p"]
Sep 18 16:19:03.318413 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:03.318251    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-79569cf898-5sd2p" podUID="22dc720e-bf48-490e-883c-fcb3225cb55f" containerName="authorino" containerID="cri-o://07bd3e788e207b563ef964aa3289482d80ce00cdb069bad518e11b8154608dbe" gracePeriod=30
Sep 18 16:19:03.572968 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:03.572938    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79569cf898-5sd2p"
Sep 18 16:19:03.723071 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:03.723038    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22dc720e-bf48-490e-883c-fcb3225cb55f-tls-cert\") pod \"22dc720e-bf48-490e-883c-fcb3225cb55f\" (UID: \"22dc720e-bf48-490e-883c-fcb3225cb55f\") "
Sep 18 16:19:03.723260 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:03.723085    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nlm6\" (UniqueName: \"kubernetes.io/projected/22dc720e-bf48-490e-883c-fcb3225cb55f-kube-api-access-9nlm6\") pod \"22dc720e-bf48-490e-883c-fcb3225cb55f\" (UID: \"22dc720e-bf48-490e-883c-fcb3225cb55f\") "
Sep 18 16:19:03.725201 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:03.725171    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dc720e-bf48-490e-883c-fcb3225cb55f-kube-api-access-9nlm6" (OuterVolumeSpecName: "kube-api-access-9nlm6") pod "22dc720e-bf48-490e-883c-fcb3225cb55f" (UID: "22dc720e-bf48-490e-883c-fcb3225cb55f"). InnerVolumeSpecName "kube-api-access-9nlm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:19:03.734418 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:03.734390    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22dc720e-bf48-490e-883c-fcb3225cb55f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "22dc720e-bf48-490e-883c-fcb3225cb55f" (UID: "22dc720e-bf48-490e-883c-fcb3225cb55f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:19:03.824452 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:03.824373    2546 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/22dc720e-bf48-490e-883c-fcb3225cb55f-tls-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:19:03.824452 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:03.824403    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nlm6\" (UniqueName: \"kubernetes.io/projected/22dc720e-bf48-490e-883c-fcb3225cb55f-kube-api-access-9nlm6\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:19:04.239695 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:04.239659    2546 generic.go:356] "Generic (PLEG): container finished" podID="22dc720e-bf48-490e-883c-fcb3225cb55f" containerID="07bd3e788e207b563ef964aa3289482d80ce00cdb069bad518e11b8154608dbe" exitCode=0
Sep 18 16:19:04.240182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:04.239736    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79569cf898-5sd2p" event={"ID":"22dc720e-bf48-490e-883c-fcb3225cb55f","Type":"ContainerDied","Data":"07bd3e788e207b563ef964aa3289482d80ce00cdb069bad518e11b8154608dbe"}
Sep 18 16:19:04.240182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:04.239779    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79569cf898-5sd2p" event={"ID":"22dc720e-bf48-490e-883c-fcb3225cb55f","Type":"ContainerDied","Data":"57f53b851786e3e6276d9b8843a2cc4ab4e938d7e36447b7878c12016f08513f"}
Sep 18 16:19:04.240182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:04.239799    2546 scope.go:117] "RemoveContainer" containerID="07bd3e788e207b563ef964aa3289482d80ce00cdb069bad518e11b8154608dbe"
Sep 18 16:19:04.240182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:04.239922    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79569cf898-5sd2p"
Sep 18 16:19:04.248645 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:04.248622    2546 scope.go:117] "RemoveContainer" containerID="07bd3e788e207b563ef964aa3289482d80ce00cdb069bad518e11b8154608dbe"
Sep 18 16:19:04.248938 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:19:04.248921    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bd3e788e207b563ef964aa3289482d80ce00cdb069bad518e11b8154608dbe\": container with ID starting with 07bd3e788e207b563ef964aa3289482d80ce00cdb069bad518e11b8154608dbe not found: ID does not exist" containerID="07bd3e788e207b563ef964aa3289482d80ce00cdb069bad518e11b8154608dbe"
Sep 18 16:19:04.249005 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:04.248946    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bd3e788e207b563ef964aa3289482d80ce00cdb069bad518e11b8154608dbe"} err="failed to get container status \"07bd3e788e207b563ef964aa3289482d80ce00cdb069bad518e11b8154608dbe\": rpc error: code = NotFound desc = could not find container \"07bd3e788e207b563ef964aa3289482d80ce00cdb069bad518e11b8154608dbe\": container with ID starting with 07bd3e788e207b563ef964aa3289482d80ce00cdb069bad518e11b8154608dbe not found: ID does not exist"
Sep 18 16:19:04.283710 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:04.283666    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79569cf898-5sd2p"]
Sep 18 16:19:04.285293 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:04.285268    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79569cf898-5sd2p"]
Sep 18 16:19:04.605417 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:04.605384    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22dc720e-bf48-490e-883c-fcb3225cb55f" path="/var/lib/kubelet/pods/22dc720e-bf48-490e-883c-fcb3225cb55f/volumes"
Sep 18 16:19:23.847308 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.847276    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"]
Sep 18 16:19:23.847750 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.847608    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22dc720e-bf48-490e-883c-fcb3225cb55f" containerName="authorino"
Sep 18 16:19:23.847750 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.847619    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="22dc720e-bf48-490e-883c-fcb3225cb55f" containerName="authorino"
Sep 18 16:19:23.847750 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.847687    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="22dc720e-bf48-490e-883c-fcb3225cb55f" containerName="authorino"
Sep 18 16:19:23.852213 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.852193    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.855641 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.855622    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fbc529-openshift-default-dockercfg-v49xq\""
Sep 18 16:19:23.871685 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.871663    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"]
Sep 18 16:19:23.898051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.897390    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-token\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.898051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.897440    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-podinfo\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.898051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.897465    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istiod-ca-cert\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.898051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.897507    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-workload-socket\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.898051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.897543    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-credential-socket\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.898051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.897590    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-workload-certs\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.898051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.897619    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-envoy\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.898051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.897641    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhb6l\" (UniqueName: \"kubernetes.io/projected/72164f1c-30e8-493b-9bd2-238b8cb49fa4-kube-api-access-zhb6l\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.898051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.897744    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-data\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.994582 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.994551    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"]
Sep 18 16:19:23.998129 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998113    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:23.998576 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998551    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-envoy\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.998576 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998579    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhb6l\" (UniqueName: \"kubernetes.io/projected/72164f1c-30e8-493b-9bd2-238b8cb49fa4-kube-api-access-zhb6l\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.998755 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998606    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-token\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:23.998755 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998641    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-data\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:23.998755 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998710    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-data\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.998755 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998742    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-workload-certs\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:23.998960 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998832    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-envoy\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:23.998960 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998878    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-token\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.998960 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998905    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-podinfo\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.998960 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998920    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istiod-ca-cert\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.998960 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998940    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-workload-socket\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:23.999292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998968    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd2dh\" (UniqueName: \"kubernetes.io/projected/055de468-2a31-4e04-b7df-4b81ca84d67f-kube-api-access-sd2dh\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:23.999292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.998997    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-podinfo\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:23.999292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.999059    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-workload-socket\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.999292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.999090    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-credential-socket\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.999292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.999103    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-data\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.999292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.999114    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-credential-socket\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:23.999582 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.999336    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055de468-2a31-4e04-b7df-4b81ca84d67f-istiod-ca-cert\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:23.999582 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.999389    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-workload-certs\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.999582 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.999402    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-credential-socket\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.999582 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.999448    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-workload-socket\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.999836 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.999756    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istiod-ca-cert\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:23.999836 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:23.999797    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-workload-certs\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:24.001638 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.001618    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-envoy\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:24.002037 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.001998    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-podinfo\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:24.003600 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:19:24.003568    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.003672 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:19:24.003583    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-worker-w0-1eec1d-openshift-default-dockercfg-srpn2\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1eec1d-openshift-default-dockercfg-srpn2\"" type="*v1.Secret"
Sep 18 16:19:24.011965 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.011941    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhb6l\" (UniqueName: \"kubernetes.io/projected/72164f1c-30e8-493b-9bd2-238b8cb49fa4-kube-api-access-zhb6l\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:24.013119 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.013017    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-token\") pod \"e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") " pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:24.058319 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.058289    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"]
Sep 18 16:19:24.100075 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.100002    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-data\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.100200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.100095    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-workload-certs\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.100200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.100126    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-envoy\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.100200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.100180    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-workload-socket\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.100358 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.100208    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd2dh\" (UniqueName: \"kubernetes.io/projected/055de468-2a31-4e04-b7df-4b81ca84d67f-kube-api-access-sd2dh\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.100452 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.100432    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-data\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.100590 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.100535    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-workload-certs\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.100718 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.100591    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-podinfo\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.100718 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.100654    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-credential-socket\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.100718 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.100680    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-workload-socket\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.100718 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.100688    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055de468-2a31-4e04-b7df-4b81ca84d67f-istiod-ca-cert\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.100925 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.100754    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-token\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.100925 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.100904    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-credential-socket\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.101465 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.101445    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055de468-2a31-4e04-b7df-4b81ca84d67f-istiod-ca-cert\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.102664 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.102635    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-envoy\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.103133 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.103112    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-podinfo\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.114802 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.114066    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"]
Sep 18 16:19:24.116422 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.116401    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd2dh\" (UniqueName: \"kubernetes.io/projected/055de468-2a31-4e04-b7df-4b81ca84d67f-kube-api-access-sd2dh\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.119742 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.119703    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.122593 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.122562    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-token\") pod \"e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") " pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:24.124313 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.124116    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4f2d8a-openshift-default-dockercfg-9tmrp\""
Sep 18 16:19:24.157646 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.157595    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"]
Sep 18 16:19:24.162853 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.162624    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:24.201961 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.201919    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk4cm\" (UniqueName: \"kubernetes.io/projected/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-kube-api-access-hk4cm\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.202114 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.201986    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-data\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.202114 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.202093    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-workload-socket\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.202228 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.202134    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istiod-ca-cert\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.202228 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.202180    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-credential-socket\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.202228 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.202219    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-podinfo\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.202375 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.202242    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-token\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.202375 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.202293    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-envoy\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.202375 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.202334    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-workload-certs\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.303806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.303758    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-credential-socket\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.303806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.303806    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-podinfo\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.304019 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.303830    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-token\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.304019 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.303860    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-envoy\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.304019 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.303889    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-workload-certs\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.304019 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.303967    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk4cm\" (UniqueName: \"kubernetes.io/projected/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-kube-api-access-hk4cm\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.304019 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.303997    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-data\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.304310 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.304079    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-workload-socket\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.304310 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.304096    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istiod-ca-cert\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.304996 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.304705    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-workload-certs\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.304996 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.304834    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istiod-ca-cert\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.304996 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.304964    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-data\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.305222 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.305139    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-workload-socket\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.305503 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.305477    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-credential-socket\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.306696 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.306674    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-podinfo\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.307667 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.307633    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-envoy\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.315962 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.315936    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-token\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.316872 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.316854    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk4cm\" (UniqueName: \"kubernetes.io/projected/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-kube-api-access-hk4cm\") pod \"e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") " pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.437396 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.437328    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:24.496321 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.496088    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"]
Sep 18 16:19:24.501170 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.501137    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:24.501253 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.501222    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:24.501253 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.501248    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:24.722132 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.722103    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"]
Sep 18 16:19:24.723704 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:19:24.723676    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ca1259_e61b_4a6d_81fd_bd67c87d0517.slice/crio-3ed9de66281246dd1b20247ed3f239d62b6300bfcfcd095d4c53136328c4a8d3 WatchSource:0}: Error finding container 3ed9de66281246dd1b20247ed3f239d62b6300bfcfcd095d4c53136328c4a8d3: Status 404 returned error can't find the container with id 3ed9de66281246dd1b20247ed3f239d62b6300bfcfcd095d4c53136328c4a8d3
Sep 18 16:19:24.726266 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.726232    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:24.726368 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.726299    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:24.726368 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.726332    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:24.933697 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.933665    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1eec1d-openshift-default-dockercfg-srpn2\""
Sep 18 16:19:24.935771 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:24.935742    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:25.223544 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.223514    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"]
Sep 18 16:19:25.225801 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:19:25.225773    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055de468_2a31_4e04_b7df_4b81ca84d67f.slice/crio-3046bf65df1e4b12bdbb70b69b7eebc41c2573ed92547d2e531ba0afd957cb85 WatchSource:0}: Error finding container 3046bf65df1e4b12bdbb70b69b7eebc41c2573ed92547d2e531ba0afd957cb85: Status 404 returned error can't find the container with id 3046bf65df1e4b12bdbb70b69b7eebc41c2573ed92547d2e531ba0afd957cb85
Sep 18 16:19:25.228062 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.228011    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:25.228170 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.228106    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:25.228170 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.228150    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:25.340200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.340167    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" event={"ID":"d9ca1259-e61b-4a6d-81fd-bd67c87d0517","Type":"ContainerStarted","Data":"81ddef6053096eeb1befd1ed11541a8862d598e1e1118951900b8a7ded514e58"}
Sep 18 16:19:25.340372 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.340225    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" event={"ID":"d9ca1259-e61b-4a6d-81fd-bd67c87d0517","Type":"ContainerStarted","Data":"3ed9de66281246dd1b20247ed3f239d62b6300bfcfcd095d4c53136328c4a8d3"}
Sep 18 16:19:25.342058 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.342016    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" event={"ID":"72164f1c-30e8-493b-9bd2-238b8cb49fa4","Type":"ContainerStarted","Data":"abb4387fcaff06156db15a56cb8549eef34b15b779a781289d96c04ac290a56b"}
Sep 18 16:19:25.342183 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.342077    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" event={"ID":"72164f1c-30e8-493b-9bd2-238b8cb49fa4","Type":"ContainerStarted","Data":"a0fc941e8bbce681eac809180b006c53f75d3b0f414d5d4cf08d59f3061b7ae6"}
Sep 18 16:19:25.343783 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.343759    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" event={"ID":"055de468-2a31-4e04-b7df-4b81ca84d67f","Type":"ContainerStarted","Data":"d57d4ab6f5c25ee68bf9ad66c2e4cc3777a219c4ee4a50922bc58c94985ec28a"}
Sep 18 16:19:25.343889 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.343809    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" event={"ID":"055de468-2a31-4e04-b7df-4b81ca84d67f","Type":"ContainerStarted","Data":"3046bf65df1e4b12bdbb70b69b7eebc41c2573ed92547d2e531ba0afd957cb85"}
Sep 18 16:19:25.364922 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.364870    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" podStartSLOduration=1.364850799 podStartE2EDuration="1.364850799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:25.362702057 +0000 UTC m=+861.271482444" watchObservedRunningTime="2026-09-18 16:19:25.364850799 +0000 UTC m=+861.273631151"
Sep 18 16:19:25.390806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.390757    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" podStartSLOduration=2.390741189 podStartE2EDuration="2.390741189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:25.389142393 +0000 UTC m=+861.297922736" watchObservedRunningTime="2026-09-18 16:19:25.390741189 +0000 UTC m=+861.299521531"
Sep 18 16:19:25.412587 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.412549    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" podStartSLOduration=2.4125354469999998 podStartE2EDuration="2.412535447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:25.411150625 +0000 UTC m=+861.319931000" watchObservedRunningTime="2026-09-18 16:19:25.412535447 +0000 UTC m=+861.321315795"
Sep 18 16:19:25.437821 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.437794    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:25.442224 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.442199    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:25.487292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.487266    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7591-55ed2c"]
Sep 18 16:19:25.491283 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.491267    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7591-55ed2c"
Sep 18 16:19:25.495277 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.495210    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7591-55ed2c"]
Sep 18 16:19:25.495417 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.495394    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6q5gq\""
Sep 18 16:19:25.515403 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.515380    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb4cn\" (UniqueName: \"kubernetes.io/projected/2dfd9a10-dcfb-40f1-9806-a8360f09e561-kube-api-access-jb4cn\") pod \"test-curl-7591-55ed2c\" (UID: \"2dfd9a10-dcfb-40f1-9806-a8360f09e561\") " pod="opendatahub/test-curl-7591-55ed2c"
Sep 18 16:19:25.616867 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.616832    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb4cn\" (UniqueName: \"kubernetes.io/projected/2dfd9a10-dcfb-40f1-9806-a8360f09e561-kube-api-access-jb4cn\") pod \"test-curl-7591-55ed2c\" (UID: \"2dfd9a10-dcfb-40f1-9806-a8360f09e561\") " pod="opendatahub/test-curl-7591-55ed2c"
Sep 18 16:19:25.625373 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.625349    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb4cn\" (UniqueName: \"kubernetes.io/projected/2dfd9a10-dcfb-40f1-9806-a8360f09e561-kube-api-access-jb4cn\") pod \"test-curl-7591-55ed2c\" (UID: \"2dfd9a10-dcfb-40f1-9806-a8360f09e561\") " pod="opendatahub/test-curl-7591-55ed2c"
Sep 18 16:19:25.802396 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.802338    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7591-55ed2c"
Sep 18 16:19:25.936103 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:25.936073    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:26.074656 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:26.074635    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7591-55ed2c"]
Sep 18 16:19:26.076461 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:19:26.076433    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dfd9a10_dcfb_40f1_9806_a8360f09e561.slice/crio-d839766e674cc9e39a991a35be8baa72ade93d57ead14ac5a84468d9edb0a915 WatchSource:0}: Error finding container d839766e674cc9e39a991a35be8baa72ade93d57ead14ac5a84468d9edb0a915: Status 404 returned error can't find the container with id d839766e674cc9e39a991a35be8baa72ade93d57ead14ac5a84468d9edb0a915
Sep 18 16:19:26.163341 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:26.163310    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:26.167703 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:26.167683    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:26.349752 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:26.349683    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7591-55ed2c" event={"ID":"2dfd9a10-dcfb-40f1-9806-a8360f09e561","Type":"ContainerStarted","Data":"d839766e674cc9e39a991a35be8baa72ade93d57ead14ac5a84468d9edb0a915"}
Sep 18 16:19:26.349877 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:26.349814    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:26.350256 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:26.350233    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:26.350760 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:26.350740    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:26.351088 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:26.351075    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:26.941729 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:26.941697    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:27.357738 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:27.357155    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7591-55ed2c" event={"ID":"2dfd9a10-dcfb-40f1-9806-a8360f09e561","Type":"ContainerStarted","Data":"5ed4e06382bbc2884942a98d8dd9d11228aa28360511b15a51d7cadb2fdac8bc"}
Sep 18 16:19:27.357738 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:27.357649    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:27.358735 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:27.358714    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:27.379193 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:27.379146    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7591-55ed2c" podStartSLOduration=1.317952119 podStartE2EDuration="2.379131944s" totalImagesPullingTime="1.061179825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:27.376230525 +0000 UTC m=+863.285010878" watchObservedRunningTime="2026-09-18 16:19:27.379131944 +0000 UTC m=+863.287912295"
Sep 18 16:19:27.578938 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:27.578902    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:19:28.134455 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:28.134364    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7591-55ed2c"]
Sep 18 16:19:28.141250 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:28.141225    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7591-55ed2c"]
Sep 18 16:19:28.267389 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:28.267355    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7591-dc8a42"]
Sep 18 16:19:28.267759 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:28.267747    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dfd9a10-dcfb-40f1-9806-a8360f09e561" containerName="test-curl-7591-55ed2c"
Sep 18 16:19:28.267799 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:28.267761    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dfd9a10-dcfb-40f1-9806-a8360f09e561" containerName="test-curl-7591-55ed2c"
Sep 18 16:19:28.267835 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:28.267826    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dfd9a10-dcfb-40f1-9806-a8360f09e561" containerName="test-curl-7591-55ed2c"
Sep 18 16:19:28.271161 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:28.271142    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7591-dc8a42"
Sep 18 16:19:28.280245 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:28.280225    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7591-dc8a42"]
Sep 18 16:19:28.341817 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:28.341789    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkj99\" (UniqueName: \"kubernetes.io/projected/cb5c29f1-ed71-46d9-8304-cb0fddf295b9-kube-api-access-dkj99\") pod \"test-curl-7591-dc8a42\" (UID: \"cb5c29f1-ed71-46d9-8304-cb0fddf295b9\") " pod="opendatahub/test-curl-7591-dc8a42"
Sep 18 16:19:28.363082 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:19:28.363052    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7591-55ed2c" container="test-curl-7591-55ed2c"
Sep 18 16:19:28.442824 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:28.442787    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkj99\" (UniqueName: \"kubernetes.io/projected/cb5c29f1-ed71-46d9-8304-cb0fddf295b9-kube-api-access-dkj99\") pod \"test-curl-7591-dc8a42\" (UID: \"cb5c29f1-ed71-46d9-8304-cb0fddf295b9\") " pod="opendatahub/test-curl-7591-dc8a42"
Sep 18 16:19:28.465042 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:28.465000    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkj99\" (UniqueName: \"kubernetes.io/projected/cb5c29f1-ed71-46d9-8304-cb0fddf295b9-kube-api-access-dkj99\") pod \"test-curl-7591-dc8a42\" (UID: \"cb5c29f1-ed71-46d9-8304-cb0fddf295b9\") " pod="opendatahub/test-curl-7591-dc8a42"
Sep 18 16:19:28.580911 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:28.580876    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7591-dc8a42"
Sep 18 16:19:28.877650 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:28.877618    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7591-dc8a42"]
Sep 18 16:19:28.881139 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:19:28.881095    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb5c29f1_ed71_46d9_8304_cb0fddf295b9.slice/crio-fce8fa408c51dbb39a70ad14829341a081d1d2e99f17e3bf0e192935f78d230c WatchSource:0}: Error finding container fce8fa408c51dbb39a70ad14829341a081d1d2e99f17e3bf0e192935f78d230c: Status 404 returned error can't find the container with id fce8fa408c51dbb39a70ad14829341a081d1d2e99f17e3bf0e192935f78d230c
Sep 18 16:19:29.374927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:29.374889    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7591-dc8a42" event={"ID":"cb5c29f1-ed71-46d9-8304-cb0fddf295b9","Type":"ContainerStarted","Data":"37c1a8949b09a77c46b8eae50d7756f2eaa271a39e719bd9e4ba3201c2bc575d"}
Sep 18 16:19:29.375359 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:29.374939    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7591-dc8a42" event={"ID":"cb5c29f1-ed71-46d9-8304-cb0fddf295b9","Type":"ContainerStarted","Data":"fce8fa408c51dbb39a70ad14829341a081d1d2e99f17e3bf0e192935f78d230c"}
Sep 18 16:19:29.375359 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:29.375334    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7591-55ed2c" podUID="2dfd9a10-dcfb-40f1-9806-a8360f09e561" containerName="test-curl-7591-55ed2c" containerID="cri-o://5ed4e06382bbc2884942a98d8dd9d11228aa28360511b15a51d7cadb2fdac8bc" gracePeriod=2
Sep 18 16:19:29.414839 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:29.414778    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7591-dc8a42" podStartSLOduration=1.414756951 podStartE2EDuration="1.414756951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:29.414420332 +0000 UTC m=+865.323200682" watchObservedRunningTime="2026-09-18 16:19:29.414756951 +0000 UTC m=+865.323537309"
Sep 18 16:19:30.266472 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:30.266399    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7591-dc8a42"]
Sep 18 16:19:30.278836 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:30.278809    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7591-dc8a42"]
Sep 18 16:19:30.379611 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:19:30.379576    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7591-dc8a42" container="test-curl-7591-dc8a42"
Sep 18 16:19:31.172294 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.172260    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"]
Sep 18 16:19:31.172827 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.172811    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5c29f1-ed71-46d9-8304-cb0fddf295b9" containerName="test-curl-7591-dc8a42"
Sep 18 16:19:31.172878 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.172832    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5c29f1-ed71-46d9-8304-cb0fddf295b9" containerName="test-curl-7591-dc8a42"
Sep 18 16:19:31.172956 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.172940    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5c29f1-ed71-46d9-8304-cb0fddf295b9" containerName="test-curl-7591-dc8a42"
Sep 18 16:19:31.176479 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.176455    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.183914 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.183893    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-eef18d-openshift-default-dockercfg-jvpff\""
Sep 18 16:19:31.194253 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.194195    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"]
Sep 18 16:19:31.272177 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.272150    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-envoy\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.272177 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.272181    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-data\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.272713 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.272227    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-podinfo\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.272713 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.272347    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-workload-certs\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.272713 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.272387    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-token\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.272713 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.272419    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-workload-socket\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.272713 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.272531    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-credential-socket\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.272713 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.272554    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6ec037-867b-4f80-a175-595eb9355fcd-istiod-ca-cert\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.272713 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.272609    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmfp8\" (UniqueName: \"kubernetes.io/projected/4f6ec037-867b-4f80-a175-595eb9355fcd-kube-api-access-gmfp8\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.373598 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.373566    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-envoy\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.373598 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.373600    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-data\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.373822 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.373633    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-podinfo\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.373822 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.373667    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-workload-certs\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.373822 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.373686    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-token\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.373822 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.373710    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-workload-socket\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.373822 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.373810    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-credential-socket\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.374110 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.373837    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6ec037-867b-4f80-a175-595eb9355fcd-istiod-ca-cert\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.374110 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.373879    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmfp8\" (UniqueName: \"kubernetes.io/projected/4f6ec037-867b-4f80-a175-595eb9355fcd-kube-api-access-gmfp8\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.374213 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.374107    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-data\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.374213 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.374156    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-workload-socket\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.374374 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.374344    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-credential-socket\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.374521 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.374500    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-workload-certs\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.374717 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.374692    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6ec037-867b-4f80-a175-595eb9355fcd-istiod-ca-cert\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.375999 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.375976    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-envoy\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.376204 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.376186    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-podinfo\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.386223 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.386195    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7591-dc8a42" podUID="cb5c29f1-ed71-46d9-8304-cb0fddf295b9" containerName="test-curl-7591-dc8a42" containerID="cri-o://37c1a8949b09a77c46b8eae50d7756f2eaa271a39e719bd9e4ba3201c2bc575d" gracePeriod=2
Sep 18 16:19:31.390977 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.390958    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmfp8\" (UniqueName: \"kubernetes.io/projected/4f6ec037-867b-4f80-a175-595eb9355fcd-kube-api-access-gmfp8\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.391272 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.391249    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-token\") pod \"e2e-autores-eef18d-openshift-default-5f67b8d696-9955f\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") " pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.494547 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.494514    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:31.538569 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.537947    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7591-55ed2c"
Sep 18 16:19:31.575966 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.575899    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb4cn\" (UniqueName: \"kubernetes.io/projected/2dfd9a10-dcfb-40f1-9806-a8360f09e561-kube-api-access-jb4cn\") pod \"2dfd9a10-dcfb-40f1-9806-a8360f09e561\" (UID: \"2dfd9a10-dcfb-40f1-9806-a8360f09e561\") "
Sep 18 16:19:31.579622 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.578890    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfd9a10-dcfb-40f1-9806-a8360f09e561-kube-api-access-jb4cn" (OuterVolumeSpecName: "kube-api-access-jb4cn") pod "2dfd9a10-dcfb-40f1-9806-a8360f09e561" (UID: "2dfd9a10-dcfb-40f1-9806-a8360f09e561"). InnerVolumeSpecName "kube-api-access-jb4cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:19:31.677870 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.677796    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb4cn\" (UniqueName: \"kubernetes.io/projected/2dfd9a10-dcfb-40f1-9806-a8360f09e561-kube-api-access-jb4cn\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:19:31.872269 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.872242    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"]
Sep 18 16:19:31.873695 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:19:31.873665    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6ec037_867b_4f80_a175_595eb9355fcd.slice/crio-8019e8813e8f5e665fd5bdd34822f3bdd6cdc5bb9d1ede15841f3cc03c01c9c3 WatchSource:0}: Error finding container 8019e8813e8f5e665fd5bdd34822f3bdd6cdc5bb9d1ede15841f3cc03c01c9c3: Status 404 returned error can't find the container with id 8019e8813e8f5e665fd5bdd34822f3bdd6cdc5bb9d1ede15841f3cc03c01c9c3
Sep 18 16:19:31.875864 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.875833    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:31.875926 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.875897    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:31.875958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:31.875926    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:32.392848 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:32.392818    2546 generic.go:356] "Generic (PLEG): container finished" podID="2dfd9a10-dcfb-40f1-9806-a8360f09e561" containerID="5ed4e06382bbc2884942a98d8dd9d11228aa28360511b15a51d7cadb2fdac8bc" exitCode=137
Sep 18 16:19:32.393243 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:32.392874    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7591-55ed2c"
Sep 18 16:19:32.393243 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:32.392921    2546 scope.go:117] "RemoveContainer" containerID="5ed4e06382bbc2884942a98d8dd9d11228aa28360511b15a51d7cadb2fdac8bc"
Sep 18 16:19:32.395261 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:32.395228    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" event={"ID":"4f6ec037-867b-4f80-a175-595eb9355fcd","Type":"ContainerStarted","Data":"aa4b7b8f285c65643518b2fc64d0c66508528a6290aed1dfda72323163f1dddb"}
Sep 18 16:19:32.395396 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:32.395287    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" event={"ID":"4f6ec037-867b-4f80-a175-595eb9355fcd","Type":"ContainerStarted","Data":"8019e8813e8f5e665fd5bdd34822f3bdd6cdc5bb9d1ede15841f3cc03c01c9c3"}
Sep 18 16:19:32.402299 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:32.402282    2546 scope.go:117] "RemoveContainer" containerID="5ed4e06382bbc2884942a98d8dd9d11228aa28360511b15a51d7cadb2fdac8bc"
Sep 18 16:19:32.402600 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:19:32.402569    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed4e06382bbc2884942a98d8dd9d11228aa28360511b15a51d7cadb2fdac8bc\": container with ID starting with 5ed4e06382bbc2884942a98d8dd9d11228aa28360511b15a51d7cadb2fdac8bc not found: ID does not exist" containerID="5ed4e06382bbc2884942a98d8dd9d11228aa28360511b15a51d7cadb2fdac8bc"
Sep 18 16:19:32.402679 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:32.402611    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed4e06382bbc2884942a98d8dd9d11228aa28360511b15a51d7cadb2fdac8bc"} err="failed to get container status \"5ed4e06382bbc2884942a98d8dd9d11228aa28360511b15a51d7cadb2fdac8bc\": rpc error: code = NotFound desc = could not find container \"5ed4e06382bbc2884942a98d8dd9d11228aa28360511b15a51d7cadb2fdac8bc\": container with ID starting with 5ed4e06382bbc2884942a98d8dd9d11228aa28360511b15a51d7cadb2fdac8bc not found: ID does not exist"
Sep 18 16:19:32.425589 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:32.425537    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" podStartSLOduration=1.425522252 podStartE2EDuration="1.425522252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:32.424680227 +0000 UTC m=+868.333460602" watchObservedRunningTime="2026-09-18 16:19:32.425522252 +0000 UTC m=+868.334302603"
Sep 18 16:19:32.429969 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:19:32.429946    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7591-55ed2c\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="2dfd9a10-dcfb-40f1-9806-a8360f09e561" pod="opendatahub/test-curl-7591-55ed2c"
Sep 18 16:19:32.494804 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:32.494780    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:32.496139 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:32.496119    2546 patch_prober.go:28] interesting pod/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 16:19:32.496215 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:32.496160    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 16:19:32.603623 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:32.603591    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dfd9a10-dcfb-40f1-9806-a8360f09e561" path="/var/lib/kubelet/pods/2dfd9a10-dcfb-40f1-9806-a8360f09e561/volumes"
Sep 18 16:19:33.500165 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:33.500136    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:33.532555 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:33.532530    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7591-dc8a42"
Sep 18 16:19:33.596778 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:33.596748    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkj99\" (UniqueName: \"kubernetes.io/projected/cb5c29f1-ed71-46d9-8304-cb0fddf295b9-kube-api-access-dkj99\") pod \"cb5c29f1-ed71-46d9-8304-cb0fddf295b9\" (UID: \"cb5c29f1-ed71-46d9-8304-cb0fddf295b9\") "
Sep 18 16:19:33.598735 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:33.598715    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb5c29f1-ed71-46d9-8304-cb0fddf295b9-kube-api-access-dkj99" (OuterVolumeSpecName: "kube-api-access-dkj99") pod "cb5c29f1-ed71-46d9-8304-cb0fddf295b9" (UID: "cb5c29f1-ed71-46d9-8304-cb0fddf295b9"). InnerVolumeSpecName "kube-api-access-dkj99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:19:33.698454 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:33.698430    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkj99\" (UniqueName: \"kubernetes.io/projected/cb5c29f1-ed71-46d9-8304-cb0fddf295b9-kube-api-access-dkj99\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:19:34.407654 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.407619    2546 generic.go:356] "Generic (PLEG): container finished" podID="cb5c29f1-ed71-46d9-8304-cb0fddf295b9" containerID="37c1a8949b09a77c46b8eae50d7756f2eaa271a39e719bd9e4ba3201c2bc575d" exitCode=137
Sep 18 16:19:34.407828 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.407672    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7591-dc8a42"
Sep 18 16:19:34.407828 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.407725    2546 scope.go:117] "RemoveContainer" containerID="37c1a8949b09a77c46b8eae50d7756f2eaa271a39e719bd9e4ba3201c2bc575d"
Sep 18 16:19:34.408103 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.408070    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:34.409140 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.409121    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:19:34.417498 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.417483    2546 scope.go:117] "RemoveContainer" containerID="37c1a8949b09a77c46b8eae50d7756f2eaa271a39e719bd9e4ba3201c2bc575d"
Sep 18 16:19:34.417758 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:19:34.417739    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c1a8949b09a77c46b8eae50d7756f2eaa271a39e719bd9e4ba3201c2bc575d\": container with ID starting with 37c1a8949b09a77c46b8eae50d7756f2eaa271a39e719bd9e4ba3201c2bc575d not found: ID does not exist" containerID="37c1a8949b09a77c46b8eae50d7756f2eaa271a39e719bd9e4ba3201c2bc575d"
Sep 18 16:19:34.417810 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.417765    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c1a8949b09a77c46b8eae50d7756f2eaa271a39e719bd9e4ba3201c2bc575d"} err="failed to get container status \"37c1a8949b09a77c46b8eae50d7756f2eaa271a39e719bd9e4ba3201c2bc575d\": rpc error: code = NotFound desc = could not find container \"37c1a8949b09a77c46b8eae50d7756f2eaa271a39e719bd9e4ba3201c2bc575d\": container with ID starting with 37c1a8949b09a77c46b8eae50d7756f2eaa271a39e719bd9e4ba3201c2bc575d not found: ID does not exist"
Sep 18 16:19:34.452877 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:19:34.452844    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7591-dc8a42\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="cb5c29f1-ed71-46d9-8304-cb0fddf295b9" pod="opendatahub/test-curl-7591-dc8a42"
Sep 18 16:19:34.601113 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:19:34.601071    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7591-dc8a42\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="cb5c29f1-ed71-46d9-8304-cb0fddf295b9" pod="opendatahub/test-curl-7591-dc8a42"
Sep 18 16:19:34.603096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.603069    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb5c29f1-ed71-46d9-8304-cb0fddf295b9" path="/var/lib/kubelet/pods/cb5c29f1-ed71-46d9-8304-cb0fddf295b9/volumes"
Sep 18 16:19:34.811811 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.811779    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"]
Sep 18 16:19:34.817138 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.817114    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:34.820551 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.820528    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-75688a34-openshift-default-dockercfg-6mx28\""
Sep 18 16:19:34.848106 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.848080    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"]
Sep 18 16:19:34.909954 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.909928    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-data\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:34.910118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.909968    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istiod-ca-cert\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:34.910118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.909990    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-workload-certs\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:34.910118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.910043    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkhz5\" (UniqueName: \"kubernetes.io/projected/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-kube-api-access-xkhz5\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:34.910118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.910076    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-workload-socket\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:34.910118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.910093    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-envoy\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:34.910283 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.910120    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-podinfo\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:34.910283 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.910154    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-token\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:34.910283 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:34.910180    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-credential-socket\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.011820 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.011770    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-data\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.012007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.011865    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istiod-ca-cert\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.012007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.011911    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-workload-certs\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.012007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.011938    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkhz5\" (UniqueName: \"kubernetes.io/projected/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-kube-api-access-xkhz5\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.012007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.011981    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-workload-socket\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.012259 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.012012    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-envoy\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.012259 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.012099    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-podinfo\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.012259 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.012123    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-token\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.012259 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.012172    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-credential-socket\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.012676 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.012644    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-credential-socket\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.012995 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.012970    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-data\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.013601 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.013575    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istiod-ca-cert\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.013856 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.013834    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-workload-certs\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.014534 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.014511    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-workload-socket\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.018486 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.018460    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-envoy\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.021799 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.021748    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-podinfo\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.025798 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.025767    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-token\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.026748 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.026727    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkhz5\" (UniqueName: \"kubernetes.io/projected/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-kube-api-access-xkhz5\") pod \"e2e-ait-75688a34-openshift-default-6b457f876-6nnh8\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") " pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.127879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.127790    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:35.473094 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.473073    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"]
Sep 18 16:19:35.475574 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:19:35.475544    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc126d1_bcf3_4804_ae72_484b7ea30d9e.slice/crio-28b45312d5f363428bdf266cca27004c8a4007026120e5d959aeea2af39856fc WatchSource:0}: Error finding container 28b45312d5f363428bdf266cca27004c8a4007026120e5d959aeea2af39856fc: Status 404 returned error can't find the container with id 28b45312d5f363428bdf266cca27004c8a4007026120e5d959aeea2af39856fc
Sep 18 16:19:35.478429 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.478385    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:35.478513 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.478475    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:35.478550 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:35.478517    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:36.424926 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:36.424882    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8" event={"ID":"4dc126d1-bcf3-4804-ae72-484b7ea30d9e","Type":"ContainerStarted","Data":"76cb861f98edec6615c67dfb63362041cbee5f391680ef5693c149f1bae98ea2"}
Sep 18 16:19:36.425440 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:36.424951    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8" event={"ID":"4dc126d1-bcf3-4804-ae72-484b7ea30d9e","Type":"ContainerStarted","Data":"28b45312d5f363428bdf266cca27004c8a4007026120e5d959aeea2af39856fc"}
Sep 18 16:19:36.468267 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:36.468202    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8" podStartSLOduration=2.4681818 podStartE2EDuration="2.4681818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:36.467304757 +0000 UTC m=+872.376085127" watchObservedRunningTime="2026-09-18 16:19:36.4681818 +0000 UTC m=+872.376962152"
Sep 18 16:19:37.128890 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:37.128846    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:37.135020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:37.134993    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:37.455170 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:37.454188    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:37.470557 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:37.470275    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:19:47.558710 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:47.558671    2546 generic.go:356] "Generic (PLEG): container finished" podID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerID="81ddef6053096eeb1befd1ed11541a8862d598e1e1118951900b8a7ded514e58" exitCode=0
Sep 18 16:19:47.559177 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:47.558745    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" event={"ID":"d9ca1259-e61b-4a6d-81fd-bd67c87d0517","Type":"ContainerDied","Data":"81ddef6053096eeb1befd1ed11541a8862d598e1e1118951900b8a7ded514e58"}
Sep 18 16:19:47.559262 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:47.559244    2546 scope.go:117] "RemoveContainer" containerID="81ddef6053096eeb1befd1ed11541a8862d598e1e1118951900b8a7ded514e58"
Sep 18 16:19:47.560665 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:47.560627    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:47.560791 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:47.560715    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:47.560791 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:47.560759    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:47.932650 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:47.932570    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"]
Sep 18 16:19:47.935013 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:47.934994    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:47.945660 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:47.945636    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f22789-openshift-default-dockercfg-4lgg6\""
Sep 18 16:19:47.962884 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:47.962856    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"]
Sep 18 16:19:48.024733 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.024701    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc23679e-5471-4e1c-8bb8-714173084201-istio-podinfo\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.024913 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.024750    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7ldn\" (UniqueName: \"kubernetes.io/projected/dc23679e-5471-4e1c-8bb8-714173084201-kube-api-access-f7ldn\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.024913 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.024769    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-istio-data\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.024913 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.024804    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-credential-socket\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.024913 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.024873    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc23679e-5471-4e1c-8bb8-714173084201-istiod-ca-cert\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.024913 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.024912    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-workload-certs\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.025197 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.024943    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-workload-socket\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.025197 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.024990    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-istio-envoy\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.025197 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.025012    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc23679e-5471-4e1c-8bb8-714173084201-istio-token\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.126268 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.126215    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc23679e-5471-4e1c-8bb8-714173084201-istiod-ca-cert\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.126460 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.126279    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-workload-certs\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.126460 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.126327    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-workload-socket\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.126460 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.126360    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-istio-envoy\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.126460 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.126391    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc23679e-5471-4e1c-8bb8-714173084201-istio-token\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.126460 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.126449    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc23679e-5471-4e1c-8bb8-714173084201-istio-podinfo\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.126738 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.126505    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7ldn\" (UniqueName: \"kubernetes.io/projected/dc23679e-5471-4e1c-8bb8-714173084201-kube-api-access-f7ldn\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.126738 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.126531    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-istio-data\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.126738 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.126567    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-credential-socket\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.127408 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.127014    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-credential-socket\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.128608 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.127609    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-istio-data\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.128608 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.127815    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-workload-certs\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.128608 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.128206    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-workload-socket\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.128608 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.128532    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc23679e-5471-4e1c-8bb8-714173084201-istiod-ca-cert\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.130118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.130067    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc23679e-5471-4e1c-8bb8-714173084201-istio-podinfo\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.131395 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.131374    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-istio-envoy\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.131823 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.131797    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:19:48.144331 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.144289    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7ldn\" (UniqueName: \"kubernetes.io/projected/dc23679e-5471-4e1c-8bb8-714173084201-kube-api-access-f7ldn\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.144862 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.144842    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc23679e-5471-4e1c-8bb8-714173084201-istio-token\") pod \"e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") " pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.247156 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.247110    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:48.567755 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.567664    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" event={"ID":"d9ca1259-e61b-4a6d-81fd-bd67c87d0517","Type":"ContainerStarted","Data":"88b1721921d924bab215bbe10c07584fbe0593ebd16265a1f1f7c9dc862e1db2"}
Sep 18 16:19:48.569451 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.569427    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"]
Sep 18 16:19:48.569545 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.569528    2546 generic.go:356] "Generic (PLEG): container finished" podID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerID="abb4387fcaff06156db15a56cb8549eef34b15b779a781289d96c04ac290a56b" exitCode=0
Sep 18 16:19:48.569649 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.569594    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" event={"ID":"72164f1c-30e8-493b-9bd2-238b8cb49fa4","Type":"ContainerDied","Data":"abb4387fcaff06156db15a56cb8549eef34b15b779a781289d96c04ac290a56b"}
Sep 18 16:19:48.570000 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.569982    2546 scope.go:117] "RemoveContainer" containerID="abb4387fcaff06156db15a56cb8549eef34b15b779a781289d96c04ac290a56b"
Sep 18 16:19:48.571706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.571674    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:48.571805 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.571745    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:48.571805 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.571779    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:48.572077 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.572055    2546 generic.go:356] "Generic (PLEG): container finished" podID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerID="d57d4ab6f5c25ee68bf9ad66c2e4cc3777a219c4ee4a50922bc58c94985ec28a" exitCode=0
Sep 18 16:19:48.572171 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.572130    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" event={"ID":"055de468-2a31-4e04-b7df-4b81ca84d67f","Type":"ContainerDied","Data":"d57d4ab6f5c25ee68bf9ad66c2e4cc3777a219c4ee4a50922bc58c94985ec28a"}
Sep 18 16:19:48.573398 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:19:48.572866    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc23679e_5471_4e1c_8bb8_714173084201.slice/crio-6779d21a009a4cab1181a36853372ea9c331dedb9336bc81602b05da4078e1a0 WatchSource:0}: Error finding container 6779d21a009a4cab1181a36853372ea9c331dedb9336bc81602b05da4078e1a0: Status 404 returned error can't find the container with id 6779d21a009a4cab1181a36853372ea9c331dedb9336bc81602b05da4078e1a0
Sep 18 16:19:48.573398 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.573267    2546 scope.go:117] "RemoveContainer" containerID="d57d4ab6f5c25ee68bf9ad66c2e4cc3777a219c4ee4a50922bc58c94985ec28a"
Sep 18 16:19:48.605861 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.605079    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:48.605861 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.605234    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:48.605861 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.605333    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:48.621181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.619974    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:48.621181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.620142    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:48.621181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:48.620229    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:19:49.437733 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:49.437698    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:49.437961 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:49.437935    2546 patch_prober.go:28] interesting pod/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 16:19:49.438071 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:49.437994    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 16:19:49.581444 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:49.581405    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" event={"ID":"72164f1c-30e8-493b-9bd2-238b8cb49fa4","Type":"ContainerStarted","Data":"bb6a62dcf07783f0e3dabafabb53568d789f4ef1690887f54fb8dcd17743a2b9"}
Sep 18 16:19:49.584406 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:49.584371    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" event={"ID":"dc23679e-5471-4e1c-8bb8-714173084201","Type":"ContainerStarted","Data":"27bffc9521e0bb03fb980379de41e5e56cb9b94ad8c77906dceac56fc32896f4"}
Sep 18 16:19:49.584530 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:49.584431    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" event={"ID":"dc23679e-5471-4e1c-8bb8-714173084201","Type":"ContainerStarted","Data":"6779d21a009a4cab1181a36853372ea9c331dedb9336bc81602b05da4078e1a0"}
Sep 18 16:19:49.587750 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:49.587721    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" event={"ID":"055de468-2a31-4e04-b7df-4b81ca84d67f","Type":"ContainerStarted","Data":"5cd4a6b31cc1495f61684c9b4215fbbd2af17c90cca8a287c9391bf015c73d3e"}
Sep 18 16:19:49.642725 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:49.642656    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" podStartSLOduration=2.642630916 podStartE2EDuration="2.642630916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:49.640662442 +0000 UTC m=+885.549442818" watchObservedRunningTime="2026-09-18 16:19:49.642630916 +0000 UTC m=+885.551411268"
Sep 18 16:19:49.937359 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:49.936941    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:49.937587 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:49.937488    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 16:19:49.937587 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:49.937550    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 16:19:50.163039 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:50.162974    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:50.163498 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:50.163464    2546 patch_prober.go:28] interesting pod/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 16:19:50.163625 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:50.163541    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 16:19:50.255103 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:50.254662    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:50.255103 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:50.254791    2546 patch_prober.go:28] interesting pod/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 18 16:19:50.255103 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:50.254836    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 18 16:19:50.441465 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:50.441422    2546 patch_prober.go:28] interesting pod/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 16:19:50.441640 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:50.441491    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 16:19:50.937140 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:50.937101    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 16:19:50.937737 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:50.937703    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 16:19:51.163646 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:51.163607    2546 patch_prober.go:28] interesting pod/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 16:19:51.163937 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:51.163910    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 16:19:51.254832 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:51.254804    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:51.438178 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:51.438138    2546 patch_prober.go:28] interesting pod/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 16:19:51.438467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:51.438441    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 16:19:51.603804 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:51.603714    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:51.607492 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:51.607457    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:19:51.936535 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:51.936434    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 16:19:51.936828 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:51.936782    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 16:19:52.163830 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:52.163789    2546 patch_prober.go:28] interesting pod/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 16:19:52.164391 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:52.164356    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 16:19:52.438678 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:52.438637    2546 patch_prober.go:28] interesting pod/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 16:19:52.438990 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:52.438959    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 16:19:52.936477 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:52.936437    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 16:19:52.936775 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:52.936743    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 16:19:53.087242 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.087211    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v"]
Sep 18 16:19:53.095774 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.095735    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v"
Sep 18 16:19:53.100381 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.100356    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:19:53.100824 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.100803    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:19:53.103931 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.103908    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v"]
Sep 18 16:19:53.166862 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.166400    2546 patch_prober.go:28] interesting pod/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 16:19:53.166862 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.166458    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 16:19:53.219752 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.219708    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387c03d-2e0e-4148-b733-c658b39245d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-75688a34-8g24v\" (UID: \"a387c03d-2e0e-4148-b733-c658b39245d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v"
Sep 18 16:19:53.320990 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.320949    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387c03d-2e0e-4148-b733-c658b39245d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-75688a34-8g24v\" (UID: \"a387c03d-2e0e-4148-b733-c658b39245d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v"
Sep 18 16:19:53.321995 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.321939    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387c03d-2e0e-4148-b733-c658b39245d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-75688a34-8g24v\" (UID: \"a387c03d-2e0e-4148-b733-c658b39245d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v"
Sep 18 16:19:53.438472 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.438423    2546 patch_prober.go:28] interesting pod/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 16:19:53.438649 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.438500    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 16:19:53.462553 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.462512    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v"
Sep 18 16:19:53.948047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.945117    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 16:19:53.948047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:53.945174    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 16:19:54.084111 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:54.083709    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v"]
Sep 18 16:19:54.111635 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:19:54.111573    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda387c03d_2e0e_4148_b733_c658b39245d2.slice/crio-890dd9e2a2de6e9d625b2e3de98e663f092226aa4e30bbc2910dbd83c871037f WatchSource:0}: Error finding container 890dd9e2a2de6e9d625b2e3de98e663f092226aa4e30bbc2910dbd83c871037f: Status 404 returned error can't find the container with id 890dd9e2a2de6e9d625b2e3de98e663f092226aa4e30bbc2910dbd83c871037f
Sep 18 16:19:54.163503 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:54.163051    2546 patch_prober.go:28] interesting pod/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 16:19:54.163503 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:54.163114    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 16:19:54.438527 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:54.438435    2546 patch_prober.go:28] interesting pod/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Sep 18 16:19:54.439107 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:54.439069    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Sep 18 16:19:54.637444 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:54.637387    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v" event={"ID":"a387c03d-2e0e-4148-b733-c658b39245d2","Type":"ContainerStarted","Data":"890dd9e2a2de6e9d625b2e3de98e663f092226aa4e30bbc2910dbd83c871037f"}
Sep 18 16:19:54.937503 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:54.937465    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 16:19:54.937781 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:54.937739    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 16:19:55.163435 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:55.163398    2546 patch_prober.go:28] interesting pod/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 16:19:55.163742 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:55.163707    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 16:19:55.731684 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:55.730304    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:55.731684 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:55.731543    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:55.735372 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:55.735313    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:19:55.939521 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:55.939488    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 16:19:55.939830 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:55.939781    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 16:19:56.163136 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:56.163049    2546 patch_prober.go:28] interesting pod/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 16:19:56.163395 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:56.163369    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 16:19:56.349951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:56.349917    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:56.937098 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:56.936402    2546 patch_prober.go:28] interesting pod/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Sep 18 16:19:56.937098 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:56.936480    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Sep 18 16:19:57.164517 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:57.164481    2546 patch_prober.go:28] interesting pod/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 18 16:19:57.164775 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:57.164746    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 18 16:19:57.360842 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:57.360811    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:58.165382 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:58.165352    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:58.166749 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:58.166724    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:19:58.172011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:58.171881    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:58.174316 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:58.174286    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:19:59.724057 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:59.724002    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v" event={"ID":"a387c03d-2e0e-4148-b733-c658b39245d2","Type":"ContainerStarted","Data":"16e5c82804a635e70c160088111d056038956519d5fde954b04ed586928adb37"}
Sep 18 16:19:59.743241 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:19:59.743176    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v" podStartSLOduration=1.549768287 podStartE2EDuration="6.743156001s" totalImagesPullingTime="5.193387714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:59.741480662 +0000 UTC m=+895.650261029" watchObservedRunningTime="2026-09-18 16:19:59.743156001 +0000 UTC m=+895.651936354"
Sep 18 16:20:00.731906 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:00.731873    2546 generic.go:356] "Generic (PLEG): container finished" podID="a387c03d-2e0e-4148-b733-c658b39245d2" containerID="16e5c82804a635e70c160088111d056038956519d5fde954b04ed586928adb37" exitCode=0
Sep 18 16:20:00.732311 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:00.731924    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v" event={"ID":"a387c03d-2e0e-4148-b733-c658b39245d2","Type":"ContainerDied","Data":"16e5c82804a635e70c160088111d056038956519d5fde954b04ed586928adb37"}
Sep 18 16:20:01.738605 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.738573    2546 generic.go:356] "Generic (PLEG): container finished" podID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerID="aa4b7b8f285c65643518b2fc64d0c66508528a6290aed1dfda72323163f1dddb" exitCode=0
Sep 18 16:20:01.739117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.738645    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" event={"ID":"4f6ec037-867b-4f80-a175-595eb9355fcd","Type":"ContainerDied","Data":"aa4b7b8f285c65643518b2fc64d0c66508528a6290aed1dfda72323163f1dddb"}
Sep 18 16:20:01.739180 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.739147    2546 scope.go:117] "RemoveContainer" containerID="aa4b7b8f285c65643518b2fc64d0c66508528a6290aed1dfda72323163f1dddb"
Sep 18 16:20:01.740394 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.740372    2546 generic.go:356] "Generic (PLEG): container finished" podID="4dc126d1-bcf3-4804-ae72-484b7ea30d9e" containerID="76cb861f98edec6615c67dfb63362041cbee5f391680ef5693c149f1bae98ea2" exitCode=0
Sep 18 16:20:01.740515 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.740414    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:01.740515 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.740439    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8" event={"ID":"4dc126d1-bcf3-4804-ae72-484b7ea30d9e","Type":"ContainerDied","Data":"76cb861f98edec6615c67dfb63362041cbee5f391680ef5693c149f1bae98ea2"}
Sep 18 16:20:01.740515 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.740481    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:01.740515 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.740514    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:01.741013 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.740959    2546 scope.go:117] "RemoveContainer" containerID="76cb861f98edec6615c67dfb63362041cbee5f391680ef5693c149f1bae98ea2"
Sep 18 16:20:01.742209 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.742174    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:01.742307 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.742248    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:01.742307 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.742295    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:01.931655 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.931627    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v"
Sep 18 16:20:01.989951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.989139    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387c03d-2e0e-4148-b733-c658b39245d2-maas-api-service-ca\") pod \"a387c03d-2e0e-4148-b733-c658b39245d2\" (UID: \"a387c03d-2e0e-4148-b733-c658b39245d2\") "
Sep 18 16:20:01.989951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:01.989797    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a387c03d-2e0e-4148-b733-c658b39245d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a387c03d-2e0e-4148-b733-c658b39245d2" (UID: "a387c03d-2e0e-4148-b733-c658b39245d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:20:02.090607 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:02.090569    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a387c03d-2e0e-4148-b733-c658b39245d2-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:02.748454 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:02.748411    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8" event={"ID":"4dc126d1-bcf3-4804-ae72-484b7ea30d9e","Type":"ContainerStarted","Data":"6bf88615c63aef34f5ead76d55519f423599cdd8df930e5202656b3439fabdac"}
Sep 18 16:20:02.750555 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:02.750527    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" event={"ID":"4f6ec037-867b-4f80-a175-595eb9355fcd","Type":"ContainerStarted","Data":"6ded38bfd83180008ea092468d2b9bea84345ad09fe0ea7cba49f7308c849486"}
Sep 18 16:20:02.751741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:02.751722    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v" event={"ID":"a387c03d-2e0e-4148-b733-c658b39245d2","Type":"ContainerDied","Data":"890dd9e2a2de6e9d625b2e3de98e663f092226aa4e30bbc2910dbd83c871037f"}
Sep 18 16:20:02.751830 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:02.751754    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="890dd9e2a2de6e9d625b2e3de98e663f092226aa4e30bbc2910dbd83c871037f"
Sep 18 16:20:02.751830 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:02.751758    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v"
Sep 18 16:20:03.128411 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:03.128371    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:20:03.129477 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:03.129446    2546 patch_prober.go:28] interesting pod/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 18 16:20:03.129602 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:03.129505    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8" podUID="4dc126d1-bcf3-4804-ae72-484b7ea30d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 18 16:20:03.499487 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:03.499453    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:20:03.499855 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:03.499833    2546 patch_prober.go:28] interesting pod/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 16:20:03.500013 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:03.499990    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 16:20:04.145527 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.144119    2546 patch_prober.go:28] interesting pod/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 18 16:20:04.145527 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.144182    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8" podUID="4dc126d1-bcf3-4804-ae72-484b7ea30d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 18 16:20:04.410819 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.410681    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:20:04.442115 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.442078    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v"]
Sep 18 16:20:04.445204 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.445176    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-75688a34-8g24v"]
Sep 18 16:20:04.497092 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.497056    2546 patch_prober.go:28] interesting pod/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 16:20:04.497370 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.497337    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 16:20:04.611063 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.606846    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"]
Sep 18 16:20:04.716703 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.716669    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a387c03d-2e0e-4148-b733-c658b39245d2" path="/var/lib/kubelet/pods/a387c03d-2e0e-4148-b733-c658b39245d2/volumes"
Sep 18 16:20:04.717668 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.717645    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"]
Sep 18 16:20:04.723421 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.723399    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a387c03d-2e0e-4148-b733-c658b39245d2" containerName="revoke-keys"
Sep 18 16:20:04.723560 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.723549    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a387c03d-2e0e-4148-b733-c658b39245d2" containerName="revoke-keys"
Sep 18 16:20:04.723747 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.723737    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a387c03d-2e0e-4148-b733-c658b39245d2" containerName="revoke-keys"
Sep 18 16:20:04.743268 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.743239    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"]
Sep 18 16:20:04.743568 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.743549    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.761052 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.757561    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4f2d8a\"/\"kube-root-ca.crt\""
Sep 18 16:20:04.761052 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.758116    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4f2d8a\"/\"simulator-w4-4f2d8a-kserve-self-signed-certs\""
Sep 18 16:20:04.761052 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.758327    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4f2d8a\"/\"default-dockercfg-h4fvj\""
Sep 18 16:20:04.761052 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.758493    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4f2d8a\"/\"openshift-service-ca.crt\""
Sep 18 16:20:04.802054 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.794166    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8" podUID="4dc126d1-bcf3-4804-ae72-484b7ea30d9e" containerName="istio-proxy" containerID="cri-o://6bf88615c63aef34f5ead76d55519f423599cdd8df930e5202656b3439fabdac" gracePeriod=30
Sep 18 16:20:04.847048 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.843579    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-dshm\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.847048 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.843633    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-tmp-dir\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.847048 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.843663    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696e445d-bd03-4d5e-b689-76d70ff3917b-tls-certs\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.847048 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.843694    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-home\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.847048 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.843770    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-model-cache\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.847048 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.843832    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jnx5\" (UniqueName: \"kubernetes.io/projected/696e445d-bd03-4d5e-b689-76d70ff3917b-kube-api-access-5jnx5\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.947045 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.945326    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jnx5\" (UniqueName: \"kubernetes.io/projected/696e445d-bd03-4d5e-b689-76d70ff3917b-kube-api-access-5jnx5\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.947045 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.945492    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-dshm\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.947045 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.945526    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-tmp-dir\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.947045 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.945557    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696e445d-bd03-4d5e-b689-76d70ff3917b-tls-certs\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.947045 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.945590    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-home\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.947045 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.945698    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-model-cache\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.947045 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.946255    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-model-cache\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.950708 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.947604    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-home\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.950708 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.947852    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-tmp-dir\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.958056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.951118    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-dshm\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.958056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.954568    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696e445d-bd03-4d5e-b689-76d70ff3917b-tls-certs\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:04.958056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:04.957108    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jnx5\" (UniqueName: \"kubernetes.io/projected/696e445d-bd03-4d5e-b689-76d70ff3917b-kube-api-access-5jnx5\") pod \"simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") " pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:05.109050 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.108243    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:05.496306 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.496268    2546 patch_prober.go:28] interesting pod/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 16:20:05.496741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.496332    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 16:20:05.700319 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.697064    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"]
Sep 18 16:20:05.712097 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.712071    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"]
Sep 18 16:20:05.712349 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.712326    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.718334 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.718312    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-50cecd8f-gw-openshift-default-dockercfg-fp4zm\""
Sep 18 16:20:05.747520 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.737351    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"]
Sep 18 16:20:05.794056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.791999    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d541e34-3351-45d2-a61d-fd3611933989-istio-podinfo\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.794056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.792078    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-workload-socket\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.794056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.792166    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-credential-socket\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.794056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.792245    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-workload-certs\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.794056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.792290    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d541e34-3351-45d2-a61d-fd3611933989-istiod-ca-cert\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.794056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.792388    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fnb5\" (UniqueName: \"kubernetes.io/projected/5d541e34-3351-45d2-a61d-fd3611933989-kube-api-access-8fnb5\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.794056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.792443    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-istio-data\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.794056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.792480    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-istio-envoy\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.794056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.792532    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d541e34-3351-45d2-a61d-fd3611933989-istio-token\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.842397 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.842339    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc" event={"ID":"696e445d-bd03-4d5e-b689-76d70ff3917b","Type":"ContainerStarted","Data":"3dca1406ec4739d1a3dc6aac88e7474de93a1e9064089460180072f8640fdaf1"}
Sep 18 16:20:05.893647 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.893605    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d541e34-3351-45d2-a61d-fd3611933989-istio-token\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.893974 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.893951    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d541e34-3351-45d2-a61d-fd3611933989-istio-podinfo\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.894165 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.894149    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-workload-socket\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.894355 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.894341    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-credential-socket\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.894541 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.894514    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-workload-certs\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.894679 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.894665    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d541e34-3351-45d2-a61d-fd3611933989-istiod-ca-cert\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.894869 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.894853    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fnb5\" (UniqueName: \"kubernetes.io/projected/5d541e34-3351-45d2-a61d-fd3611933989-kube-api-access-8fnb5\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.895010 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.894995    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-istio-data\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.895155 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.895140    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-istio-envoy\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.896254 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.896222    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-workload-certs\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.897819 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.897468    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-istio-data\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.898480 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.898184    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d541e34-3351-45d2-a61d-fd3611933989-istiod-ca-cert\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.898480 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.898444    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-workload-socket\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.898893 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.898871    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-credential-socket\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.905491 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.905289    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-istio-envoy\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.908116 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.907754    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d541e34-3351-45d2-a61d-fd3611933989-istio-token\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.910950 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.908645    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d541e34-3351-45d2-a61d-fd3611933989-istio-podinfo\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:05.911854 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:05.911820    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fnb5\" (UniqueName: \"kubernetes.io/projected/5d541e34-3351-45d2-a61d-fd3611933989-kube-api-access-8fnb5\") pod \"e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") " pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:06.065371 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:06.065132    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:06.498054 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:06.497393    2546 patch_prober.go:28] interesting pod/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 16:20:06.498054 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:06.497460    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 16:20:06.546654 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:06.546581    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"]
Sep 18 16:20:06.562224 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:06.562177    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:06.562328 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:06.562265    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:06.562328 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:06.562310    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:06.859117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:06.857303    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" event={"ID":"5d541e34-3351-45d2-a61d-fd3611933989","Type":"ContainerStarted","Data":"33021c135d571a34db677bc9ca404eb2a17819d53c0a5a64fbde7150ef51e0ae"}
Sep 18 16:20:06.859117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:06.857382    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" event={"ID":"5d541e34-3351-45d2-a61d-fd3611933989","Type":"ContainerStarted","Data":"53fedc5cc73ec3a3ece8d9c33c2e6873cf683094f2071e407e9ebd4462f42980"}
Sep 18 16:20:06.870056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:06.864504    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc" event={"ID":"696e445d-bd03-4d5e-b689-76d70ff3917b","Type":"ContainerStarted","Data":"9e77605415e05f6821bf8baf529dfbb73c2e82c5e27004d8b129fe1ad9c0a311"}
Sep 18 16:20:06.892204 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:06.890134    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" podStartSLOduration=1.8901136809999999 podStartE2EDuration="1.890113681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:06.888463741 +0000 UTC m=+902.797244110" watchObservedRunningTime="2026-09-18 16:20:06.890113681 +0000 UTC m=+902.798894033"
Sep 18 16:20:07.067837 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:07.067801    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:07.068189 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:07.068164    2546 patch_prober.go:28] interesting pod/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 18 16:20:07.068376 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:07.068350    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" podUID="5d541e34-3351-45d2-a61d-fd3611933989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 18 16:20:07.454517 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:07.454473    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:20:07.495504 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:07.495460    2546 patch_prober.go:28] interesting pod/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 16:20:07.495693 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:07.495530    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 16:20:08.066486 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:08.066455    2546 patch_prober.go:28] interesting pod/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 18 16:20:08.066944 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:08.066914    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" podUID="5d541e34-3351-45d2-a61d-fd3611933989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 18 16:20:08.525612 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:08.525580    2546 patch_prober.go:28] interesting pod/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 18 16:20:08.525904 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:08.525881    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 18 16:20:09.070398 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:09.070367    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:09.114676 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:09.114622    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc" podStartSLOduration=5.114606908 podStartE2EDuration="5.114606908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:06.925791477 +0000 UTC m=+902.834571845" watchObservedRunningTime="2026-09-18 16:20:09.114606908 +0000 UTC m=+905.023387260"
Sep 18 16:20:09.502754 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:09.501463    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:20:09.505339 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:09.505312    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:20:09.902730 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:09.902642    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:09.906045 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:09.905557    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:13.774156 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.774134    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:20:13.907704 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.907628    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-workload-certs\") pod \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") "
Sep 18 16:20:13.907704 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.907660    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istiod-ca-cert\") pod \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") "
Sep 18 16:20:13.907704 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.907682    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-data\") pod \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") "
Sep 18 16:20:13.907704 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.907698    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-envoy\") pod \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") "
Sep 18 16:20:13.908071 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.907717    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-token\") pod \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") "
Sep 18 16:20:13.908071 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.907743    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkhz5\" (UniqueName: \"kubernetes.io/projected/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-kube-api-access-xkhz5\") pod \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") "
Sep 18 16:20:13.908071 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.907792    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-credential-socket\") pod \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") "
Sep 18 16:20:13.908071 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.907846    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-podinfo\") pod \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") "
Sep 18 16:20:13.908071 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.907895    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-workload-socket\") pod \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\" (UID: \"4dc126d1-bcf3-4804-ae72-484b7ea30d9e\") "
Sep 18 16:20:13.908071 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.907924    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-data" (OuterVolumeSpecName: "istio-data") pod "4dc126d1-bcf3-4804-ae72-484b7ea30d9e" (UID: "4dc126d1-bcf3-4804-ae72-484b7ea30d9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:20:13.908071 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.907950    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4dc126d1-bcf3-4804-ae72-484b7ea30d9e" (UID: "4dc126d1-bcf3-4804-ae72-484b7ea30d9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:20:13.908441 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.908084    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4dc126d1-bcf3-4804-ae72-484b7ea30d9e" (UID: "4dc126d1-bcf3-4804-ae72-484b7ea30d9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:20:13.908441 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.908170    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4dc126d1-bcf3-4804-ae72-484b7ea30d9e" (UID: "4dc126d1-bcf3-4804-ae72-484b7ea30d9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:20:13.908441 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.908206    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4dc126d1-bcf3-4804-ae72-484b7ea30d9e" (UID: "4dc126d1-bcf3-4804-ae72-484b7ea30d9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:20:13.908441 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.908419    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-workload-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:13.908441 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.908441    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istiod-ca-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:13.908691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.908455    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-data\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:13.908691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.908467    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-credential-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:13.908691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.908480    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-workload-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:13.910005 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.909981    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-token" (OuterVolumeSpecName: "istio-token") pod "4dc126d1-bcf3-4804-ae72-484b7ea30d9e" (UID: "4dc126d1-bcf3-4804-ae72-484b7ea30d9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:20:13.910147 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.910017    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4dc126d1-bcf3-4804-ae72-484b7ea30d9e" (UID: "4dc126d1-bcf3-4804-ae72-484b7ea30d9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:20:13.910147 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.910059    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4dc126d1-bcf3-4804-ae72-484b7ea30d9e" (UID: "4dc126d1-bcf3-4804-ae72-484b7ea30d9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:20:13.910249 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.910164    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-kube-api-access-xkhz5" (OuterVolumeSpecName: "kube-api-access-xkhz5") pod "4dc126d1-bcf3-4804-ae72-484b7ea30d9e" (UID: "4dc126d1-bcf3-4804-ae72-484b7ea30d9e"). InnerVolumeSpecName "kube-api-access-xkhz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:20:13.926189 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.926167    2546 generic.go:356] "Generic (PLEG): container finished" podID="4dc126d1-bcf3-4804-ae72-484b7ea30d9e" containerID="6bf88615c63aef34f5ead76d55519f423599cdd8df930e5202656b3439fabdac" exitCode=0
Sep 18 16:20:13.926297 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.926232    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"
Sep 18 16:20:13.926297 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.926249    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8" event={"ID":"4dc126d1-bcf3-4804-ae72-484b7ea30d9e","Type":"ContainerDied","Data":"6bf88615c63aef34f5ead76d55519f423599cdd8df930e5202656b3439fabdac"}
Sep 18 16:20:13.926383 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.926346    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8" event={"ID":"4dc126d1-bcf3-4804-ae72-484b7ea30d9e","Type":"ContainerDied","Data":"28b45312d5f363428bdf266cca27004c8a4007026120e5d959aeea2af39856fc"}
Sep 18 16:20:13.926427 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.926380    2546 scope.go:117] "RemoveContainer" containerID="6bf88615c63aef34f5ead76d55519f423599cdd8df930e5202656b3439fabdac"
Sep 18 16:20:13.928091 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.928072    2546 generic.go:356] "Generic (PLEG): container finished" podID="dc23679e-5471-4e1c-8bb8-714173084201" containerID="27bffc9521e0bb03fb980379de41e5e56cb9b94ad8c77906dceac56fc32896f4" exitCode=0
Sep 18 16:20:13.928195 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.928151    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" event={"ID":"dc23679e-5471-4e1c-8bb8-714173084201","Type":"ContainerDied","Data":"27bffc9521e0bb03fb980379de41e5e56cb9b94ad8c77906dceac56fc32896f4"}
Sep 18 16:20:13.928661 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.928643    2546 scope.go:117] "RemoveContainer" containerID="27bffc9521e0bb03fb980379de41e5e56cb9b94ad8c77906dceac56fc32896f4"
Sep 18 16:20:13.930009 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.929974    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:13.930109 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.930090    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:13.930153 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.930143    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:13.936213 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.936198    2546 scope.go:117] "RemoveContainer" containerID="76cb861f98edec6615c67dfb63362041cbee5f391680ef5693c149f1bae98ea2"
Sep 18 16:20:13.945219 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.945201    2546 scope.go:117] "RemoveContainer" containerID="6bf88615c63aef34f5ead76d55519f423599cdd8df930e5202656b3439fabdac"
Sep 18 16:20:13.945498 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:20:13.945481    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf88615c63aef34f5ead76d55519f423599cdd8df930e5202656b3439fabdac\": container with ID starting with 6bf88615c63aef34f5ead76d55519f423599cdd8df930e5202656b3439fabdac not found: ID does not exist" containerID="6bf88615c63aef34f5ead76d55519f423599cdd8df930e5202656b3439fabdac"
Sep 18 16:20:13.945618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.945509    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf88615c63aef34f5ead76d55519f423599cdd8df930e5202656b3439fabdac"} err="failed to get container status \"6bf88615c63aef34f5ead76d55519f423599cdd8df930e5202656b3439fabdac\": rpc error: code = NotFound desc = could not find container \"6bf88615c63aef34f5ead76d55519f423599cdd8df930e5202656b3439fabdac\": container with ID starting with 6bf88615c63aef34f5ead76d55519f423599cdd8df930e5202656b3439fabdac not found: ID does not exist"
Sep 18 16:20:13.945618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.945531    2546 scope.go:117] "RemoveContainer" containerID="76cb861f98edec6615c67dfb63362041cbee5f391680ef5693c149f1bae98ea2"
Sep 18 16:20:13.945781 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:20:13.945759    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76cb861f98edec6615c67dfb63362041cbee5f391680ef5693c149f1bae98ea2\": container with ID starting with 76cb861f98edec6615c67dfb63362041cbee5f391680ef5693c149f1bae98ea2 not found: ID does not exist" containerID="76cb861f98edec6615c67dfb63362041cbee5f391680ef5693c149f1bae98ea2"
Sep 18 16:20:13.945820 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.945791    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76cb861f98edec6615c67dfb63362041cbee5f391680ef5693c149f1bae98ea2"} err="failed to get container status \"76cb861f98edec6615c67dfb63362041cbee5f391680ef5693c149f1bae98ea2\": rpc error: code = NotFound desc = could not find container \"76cb861f98edec6615c67dfb63362041cbee5f391680ef5693c149f1bae98ea2\": container with ID starting with 76cb861f98edec6615c67dfb63362041cbee5f391680ef5693c149f1bae98ea2 not found: ID does not exist"
Sep 18 16:20:13.996151 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.996109    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"]
Sep 18 16:20:13.999406 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:13.999382    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-75688a34-openshift-default-6b457f876-6nnh8"]
Sep 18 16:20:14.009674 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:14.009539    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-envoy\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:14.009674 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:14.009566    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-token\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:14.009674 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:14.009578    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkhz5\" (UniqueName: \"kubernetes.io/projected/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-kube-api-access-xkhz5\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:14.009674 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:14.009590    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc126d1-bcf3-4804-ae72-484b7ea30d9e-istio-podinfo\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:14.135370 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:14.135329    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:20:14.606499 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:14.606462    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dc126d1-bcf3-4804-ae72-484b7ea30d9e" path="/var/lib/kubelet/pods/4dc126d1-bcf3-4804-ae72-484b7ea30d9e/volumes"
Sep 18 16:20:14.936508 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:14.936423    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" event={"ID":"dc23679e-5471-4e1c-8bb8-714173084201","Type":"ContainerStarted","Data":"3412385dde8196da5af73dc41bc60a8e3a4eab080fb951d9afc6e397b341e67b"}
Sep 18 16:20:15.109542 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:15.109499    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:15.109542 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:15.109551    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:15.121939 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:15.121908    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:15.247942 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:15.247907    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:20:15.248119 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:15.248053    2546 patch_prober.go:28] interesting pod/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 18 16:20:15.248119 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:15.248100    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 18 16:20:15.954007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:15.953978    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:20:16.248656 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:16.248573    2546 patch_prober.go:28] interesting pod/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 18 16:20:16.248656 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:16.248644    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 18 16:20:17.248637 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:17.248604    2546 patch_prober.go:28] interesting pod/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 18 16:20:17.248979 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:17.248655    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 18 16:20:18.248715 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:18.248676    2546 patch_prober.go:28] interesting pod/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 18 16:20:18.249236 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:18.248759    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 18 16:20:19.248655 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:19.248614    2546 patch_prober.go:28] interesting pod/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 18 16:20:19.248865 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:19.248686    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 18 16:20:20.253189 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:20.253150    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:20:20.253610 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:20.253414    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:20:20.254067 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:20.254012    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:20:28.332788 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.332746    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg"]
Sep 18 16:20:28.333258 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.333242    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc126d1-bcf3-4804-ae72-484b7ea30d9e" containerName="istio-proxy"
Sep 18 16:20:28.333258 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.333259    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc126d1-bcf3-4804-ae72-484b7ea30d9e" containerName="istio-proxy"
Sep 18 16:20:28.333331 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.333267    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dc126d1-bcf3-4804-ae72-484b7ea30d9e" containerName="istio-proxy"
Sep 18 16:20:28.333331 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.333273    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dc126d1-bcf3-4804-ae72-484b7ea30d9e" containerName="istio-proxy"
Sep 18 16:20:28.333392 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.333341    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc126d1-bcf3-4804-ae72-484b7ea30d9e" containerName="istio-proxy"
Sep 18 16:20:28.333392 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.333348    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dc126d1-bcf3-4804-ae72-484b7ea30d9e" containerName="istio-proxy"
Sep 18 16:20:28.338575 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.338557    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg"
Sep 18 16:20:28.342213 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.342192    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:20:28.342355 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.342338    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:20:28.351189 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.351167    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg"]
Sep 18 16:20:28.360683 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.360636    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe53908f-e70d-4f97-aa5d-7a7b20236b4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg\" (UID: \"fe53908f-e70d-4f97-aa5d-7a7b20236b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg"
Sep 18 16:20:28.462052 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.461996    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe53908f-e70d-4f97-aa5d-7a7b20236b4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg\" (UID: \"fe53908f-e70d-4f97-aa5d-7a7b20236b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg"
Sep 18 16:20:28.462628 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.462608    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe53908f-e70d-4f97-aa5d-7a7b20236b4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg\" (UID: \"fe53908f-e70d-4f97-aa5d-7a7b20236b4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg"
Sep 18 16:20:28.649637 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.649555    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg"
Sep 18 16:20:28.953954 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:28.953922    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg"]
Sep 18 16:20:28.957142 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:20:28.957115    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe53908f_e70d_4f97_aa5d_7a7b20236b4e.slice/crio-74443b5a1e19c0aeac4f70ebcd74344be3eec91e1b6a4b08722ed8d5f85bf26a WatchSource:0}: Error finding container 74443b5a1e19c0aeac4f70ebcd74344be3eec91e1b6a4b08722ed8d5f85bf26a: Status 404 returned error can't find the container with id 74443b5a1e19c0aeac4f70ebcd74344be3eec91e1b6a4b08722ed8d5f85bf26a
Sep 18 16:20:29.036378 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:29.036342    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg" event={"ID":"fe53908f-e70d-4f97-aa5d-7a7b20236b4e","Type":"ContainerStarted","Data":"74443b5a1e19c0aeac4f70ebcd74344be3eec91e1b6a4b08722ed8d5f85bf26a"}
Sep 18 16:20:30.043794 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:30.043758    2546 generic.go:356] "Generic (PLEG): container finished" podID="fe53908f-e70d-4f97-aa5d-7a7b20236b4e" containerID="36a9aeb3762007fc692b793cc47bb4b7d9a1fdaf750ea546e347750e6113e433" exitCode=0
Sep 18 16:20:30.044210 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:30.043830    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg" event={"ID":"fe53908f-e70d-4f97-aa5d-7a7b20236b4e","Type":"ContainerDied","Data":"36a9aeb3762007fc692b793cc47bb4b7d9a1fdaf750ea546e347750e6113e433"}
Sep 18 16:20:31.200221 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:31.200191    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg"
Sep 18 16:20:31.293999 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:31.293965    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe53908f-e70d-4f97-aa5d-7a7b20236b4e-maas-api-service-ca\") pod \"fe53908f-e70d-4f97-aa5d-7a7b20236b4e\" (UID: \"fe53908f-e70d-4f97-aa5d-7a7b20236b4e\") "
Sep 18 16:20:31.294328 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:31.294308    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe53908f-e70d-4f97-aa5d-7a7b20236b4e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe53908f-e70d-4f97-aa5d-7a7b20236b4e" (UID: "fe53908f-e70d-4f97-aa5d-7a7b20236b4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:20:31.395304 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:31.395229    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe53908f-e70d-4f97-aa5d-7a7b20236b4e-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:31.594218 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:31.594182    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:20:32.063000 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:32.062969    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg"
Sep 18 16:20:32.063188 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:32.063002    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg" event={"ID":"fe53908f-e70d-4f97-aa5d-7a7b20236b4e","Type":"ContainerDied","Data":"74443b5a1e19c0aeac4f70ebcd74344be3eec91e1b6a4b08722ed8d5f85bf26a"}
Sep 18 16:20:32.063188 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:32.063095    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74443b5a1e19c0aeac4f70ebcd74344be3eec91e1b6a4b08722ed8d5f85bf26a"
Sep 18 16:20:33.069940 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:33.069905    2546 generic.go:356] "Generic (PLEG): container finished" podID="5d541e34-3351-45d2-a61d-fd3611933989" containerID="33021c135d571a34db677bc9ca404eb2a17819d53c0a5a64fbde7150ef51e0ae" exitCode=0
Sep 18 16:20:33.070359 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:33.069948    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" event={"ID":"5d541e34-3351-45d2-a61d-fd3611933989","Type":"ContainerDied","Data":"33021c135d571a34db677bc9ca404eb2a17819d53c0a5a64fbde7150ef51e0ae"}
Sep 18 16:20:33.070402 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:33.070356    2546 scope.go:117] "RemoveContainer" containerID="33021c135d571a34db677bc9ca404eb2a17819d53c0a5a64fbde7150ef51e0ae"
Sep 18 16:20:33.071534 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:33.071507    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:33.071598 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:33.071575    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:33.071637 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:33.071606    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:34.077950 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:34.077906    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" event={"ID":"5d541e34-3351-45d2-a61d-fd3611933989","Type":"ContainerStarted","Data":"59d039308f8e73fbffbe4577ec730c5c6bb4adec197d9304eadd4292bdc26a85"}
Sep 18 16:20:35.066373 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:35.066245    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:35.066642 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:35.066620    2546 patch_prober.go:28] interesting pod/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 18 16:20:35.066746 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:35.066673    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" podUID="5d541e34-3351-45d2-a61d-fd3611933989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 18 16:20:36.066643 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:36.066612    2546 patch_prober.go:28] interesting pod/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 18 16:20:36.067003 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:36.066678    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" podUID="5d541e34-3351-45d2-a61d-fd3611933989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 18 16:20:37.066302 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:37.066262    2546 patch_prober.go:28] interesting pod/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 18 16:20:37.066500 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:37.066342    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" podUID="5d541e34-3351-45d2-a61d-fd3611933989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 18 16:20:38.066018 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:38.065982    2546 patch_prober.go:28] interesting pod/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 18 16:20:38.066416 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:38.066077    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" podUID="5d541e34-3351-45d2-a61d-fd3611933989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 18 16:20:39.069737 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:39.069711    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:39.070115 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:39.070059    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:39.070629 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:39.070611    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:39.531171 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:39.531121    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg"]
Sep 18 16:20:39.534931 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:39.534895    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-50cecd8f-5hjmg"]
Sep 18 16:20:39.725626 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:39.725594    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"]
Sep 18 16:20:40.121811 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.121777    2546 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" secret="" err="secret \"e2e-migrate-50cecd8f-gw-openshift-default-dockercfg-fp4zm\" not found"
Sep 18 16:20:40.604516 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.604486    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe53908f-e70d-4f97-aa5d-7a7b20236b4e" path="/var/lib/kubelet/pods/fe53908f-e70d-4f97-aa5d-7a7b20236b4e/volumes"
Sep 18 16:20:40.703269 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.703240    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"]
Sep 18 16:20:40.703631 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.703618    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe53908f-e70d-4f97-aa5d-7a7b20236b4e" containerName="revoke-keys"
Sep 18 16:20:40.703682 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.703634    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe53908f-e70d-4f97-aa5d-7a7b20236b4e" containerName="revoke-keys"
Sep 18 16:20:40.703717 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.703702    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe53908f-e70d-4f97-aa5d-7a7b20236b4e" containerName="revoke-keys"
Sep 18 16:20:40.706091 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.706075    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.710938 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.710906    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-48de7216-openshift-default-dockercfg-mbdh6\""
Sep 18 16:20:40.743199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.743175    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"]
Sep 18 16:20:40.796883 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.796858    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-workload-socket\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.797017 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.796915    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f827731d-db49-4d99-b566-900a178d7052-istiod-ca-cert\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.797017 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.796938    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px5bw\" (UniqueName: \"kubernetes.io/projected/f827731d-db49-4d99-b566-900a178d7052-kube-api-access-px5bw\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.797017 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.796968    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f827731d-db49-4d99-b566-900a178d7052-istio-token\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.797017 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.796982    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-credential-socket\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.797017 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.797011    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-workload-certs\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.797242 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.797055    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f827731d-db49-4d99-b566-900a178d7052-istio-podinfo\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.797242 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.797084    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-istio-data\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.797242 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.797126    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-istio-envoy\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.898136 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.898049    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f827731d-db49-4d99-b566-900a178d7052-istio-podinfo\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.898136 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.898115    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-istio-data\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.898438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.898153    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-istio-envoy\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.898438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.898209    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-workload-socket\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.898438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.898271    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f827731d-db49-4d99-b566-900a178d7052-istiod-ca-cert\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.898438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.898302    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px5bw\" (UniqueName: \"kubernetes.io/projected/f827731d-db49-4d99-b566-900a178d7052-kube-api-access-px5bw\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.898438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.898357    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f827731d-db49-4d99-b566-900a178d7052-istio-token\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.898438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.898379    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-credential-socket\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.898438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.898410    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-workload-certs\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.898796 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.898646    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-istio-data\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.899092 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.899039    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f827731d-db49-4d99-b566-900a178d7052-istiod-ca-cert\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.899219 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.899168    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-workload-certs\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.899282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.899240    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-workload-socket\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.899282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.899252    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-credential-socket\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.900390 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.900370    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-istio-envoy\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.900635 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.900618    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f827731d-db49-4d99-b566-900a178d7052-istio-podinfo\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.953608 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.953584    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px5bw\" (UniqueName: \"kubernetes.io/projected/f827731d-db49-4d99-b566-900a178d7052-kube-api-access-px5bw\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:40.953696 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:40.953585    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f827731d-db49-4d99-b566-900a178d7052-istio-token\") pod \"e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") " pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:41.015798 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:41.015778    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:41.127012 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:41.126967    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" podUID="5d541e34-3351-45d2-a61d-fd3611933989" containerName="istio-proxy" containerID="cri-o://59d039308f8e73fbffbe4577ec730c5c6bb4adec197d9304eadd4292bdc26a85" gracePeriod=30
Sep 18 16:20:41.322354 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:20:41.322310    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf827731d_db49_4d99_b566_900a178d7052.slice/crio-96a848be9be7e9d99a16edb7744295ffbfba9816a5e72eda0a7ad1ac762c0fba WatchSource:0}: Error finding container 96a848be9be7e9d99a16edb7744295ffbfba9816a5e72eda0a7ad1ac762c0fba: Status 404 returned error can't find the container with id 96a848be9be7e9d99a16edb7744295ffbfba9816a5e72eda0a7ad1ac762c0fba
Sep 18 16:20:41.324869 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:41.324833    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:41.324958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:41.324902    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:41.324958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:41.324931    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:20:41.325635 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:41.325607    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"]
Sep 18 16:20:42.133984 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:42.133944    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" event={"ID":"f827731d-db49-4d99-b566-900a178d7052","Type":"ContainerStarted","Data":"649df7c629f6024ec87bad26b51ff82a28c952ab381ba5296932cc4a55234d7c"}
Sep 18 16:20:42.134390 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:42.134007    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" event={"ID":"f827731d-db49-4d99-b566-900a178d7052","Type":"ContainerStarted","Data":"96a848be9be7e9d99a16edb7744295ffbfba9816a5e72eda0a7ad1ac762c0fba"}
Sep 18 16:20:42.186341 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:42.186293    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" podStartSLOduration=2.186278785 podStartE2EDuration="2.186278785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:42.178346802 +0000 UTC m=+938.087127172" watchObservedRunningTime="2026-09-18 16:20:42.186278785 +0000 UTC m=+938.095059206"
Sep 18 16:20:43.016367 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:43.016330    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:43.021235 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:43.021212    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:43.140049 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:43.140000    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:43.140836 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:43.140817    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:20:46.289179 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.289144    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:20:46.375644 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.375621    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:46.449265 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449236    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-workload-socket\") pod \"5d541e34-3351-45d2-a61d-fd3611933989\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") "
Sep 18 16:20:46.449415 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449271    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-istio-data\") pod \"5d541e34-3351-45d2-a61d-fd3611933989\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") "
Sep 18 16:20:46.449415 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449298    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d541e34-3351-45d2-a61d-fd3611933989-istio-token\") pod \"5d541e34-3351-45d2-a61d-fd3611933989\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") "
Sep 18 16:20:46.449415 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449334    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-istio-envoy\") pod \"5d541e34-3351-45d2-a61d-fd3611933989\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") "
Sep 18 16:20:46.449415 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449397    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d541e34-3351-45d2-a61d-fd3611933989-istiod-ca-cert\") pod \"5d541e34-3351-45d2-a61d-fd3611933989\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") "
Sep 18 16:20:46.449631 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449429    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d541e34-3351-45d2-a61d-fd3611933989-istio-podinfo\") pod \"5d541e34-3351-45d2-a61d-fd3611933989\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") "
Sep 18 16:20:46.449631 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449475    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-credential-socket\") pod \"5d541e34-3351-45d2-a61d-fd3611933989\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") "
Sep 18 16:20:46.449631 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449513    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d541e34-3351-45d2-a61d-fd3611933989" (UID: "5d541e34-3351-45d2-a61d-fd3611933989"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:20:46.449631 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449520    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fnb5\" (UniqueName: \"kubernetes.io/projected/5d541e34-3351-45d2-a61d-fd3611933989-kube-api-access-8fnb5\") pod \"5d541e34-3351-45d2-a61d-fd3611933989\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") "
Sep 18 16:20:46.449631 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449571    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-workload-certs\") pod \"5d541e34-3351-45d2-a61d-fd3611933989\" (UID: \"5d541e34-3351-45d2-a61d-fd3611933989\") "
Sep 18 16:20:46.449873 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449732    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d541e34-3351-45d2-a61d-fd3611933989" (UID: "5d541e34-3351-45d2-a61d-fd3611933989"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:20:46.449873 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449770    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d541e34-3351-45d2-a61d-fd3611933989-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d541e34-3351-45d2-a61d-fd3611933989" (UID: "5d541e34-3351-45d2-a61d-fd3611933989"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:20:46.449873 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449849    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d541e34-3351-45d2-a61d-fd3611933989" (UID: "5d541e34-3351-45d2-a61d-fd3611933989"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:20:46.450123 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.449934    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d541e34-3351-45d2-a61d-fd3611933989" (UID: "5d541e34-3351-45d2-a61d-fd3611933989"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:20:46.450200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.450142    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d541e34-3351-45d2-a61d-fd3611933989-istiod-ca-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:46.450200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.450163    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-credential-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:46.450200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.450177    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-workload-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:46.450200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.450192    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-workload-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:46.450411 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.450205    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-istio-data\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:46.451641 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.451613    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d541e34-3351-45d2-a61d-fd3611933989-kube-api-access-8fnb5" (OuterVolumeSpecName: "kube-api-access-8fnb5") pod "5d541e34-3351-45d2-a61d-fd3611933989" (UID: "5d541e34-3351-45d2-a61d-fd3611933989"). InnerVolumeSpecName "kube-api-access-8fnb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:20:46.451762 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.451745    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d541e34-3351-45d2-a61d-fd3611933989-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d541e34-3351-45d2-a61d-fd3611933989" (UID: "5d541e34-3351-45d2-a61d-fd3611933989"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:20:46.452000 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.451979    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d541e34-3351-45d2-a61d-fd3611933989" (UID: "5d541e34-3351-45d2-a61d-fd3611933989"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:20:46.452087 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.452048    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d541e34-3351-45d2-a61d-fd3611933989-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d541e34-3351-45d2-a61d-fd3611933989" (UID: "5d541e34-3351-45d2-a61d-fd3611933989"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:20:46.550882 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.550856    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d541e34-3351-45d2-a61d-fd3611933989-istio-token\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:46.550882 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.550878    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d541e34-3351-45d2-a61d-fd3611933989-istio-envoy\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:46.551055 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.550895    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d541e34-3351-45d2-a61d-fd3611933989-istio-podinfo\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:46.551055 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:46.550906    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fnb5\" (UniqueName: \"kubernetes.io/projected/5d541e34-3351-45d2-a61d-fd3611933989-kube-api-access-8fnb5\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:20:47.161892 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:47.161857    2546 generic.go:356] "Generic (PLEG): container finished" podID="5d541e34-3351-45d2-a61d-fd3611933989" containerID="59d039308f8e73fbffbe4577ec730c5c6bb4adec197d9304eadd4292bdc26a85" exitCode=0
Sep 18 16:20:47.162097 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:47.161904    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"
Sep 18 16:20:47.162097 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:47.161949    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" event={"ID":"5d541e34-3351-45d2-a61d-fd3611933989","Type":"ContainerDied","Data":"59d039308f8e73fbffbe4577ec730c5c6bb4adec197d9304eadd4292bdc26a85"}
Sep 18 16:20:47.162097 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:47.162003    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx" event={"ID":"5d541e34-3351-45d2-a61d-fd3611933989","Type":"ContainerDied","Data":"53fedc5cc73ec3a3ece8d9c33c2e6873cf683094f2071e407e9ebd4462f42980"}
Sep 18 16:20:47.162097 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:47.162054    2546 scope.go:117] "RemoveContainer" containerID="59d039308f8e73fbffbe4577ec730c5c6bb4adec197d9304eadd4292bdc26a85"
Sep 18 16:20:47.170753 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:47.170738    2546 scope.go:117] "RemoveContainer" containerID="33021c135d571a34db677bc9ca404eb2a17819d53c0a5a64fbde7150ef51e0ae"
Sep 18 16:20:47.178528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:47.178466    2546 scope.go:117] "RemoveContainer" containerID="59d039308f8e73fbffbe4577ec730c5c6bb4adec197d9304eadd4292bdc26a85"
Sep 18 16:20:47.178763 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:20:47.178742    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d039308f8e73fbffbe4577ec730c5c6bb4adec197d9304eadd4292bdc26a85\": container with ID starting with 59d039308f8e73fbffbe4577ec730c5c6bb4adec197d9304eadd4292bdc26a85 not found: ID does not exist" containerID="59d039308f8e73fbffbe4577ec730c5c6bb4adec197d9304eadd4292bdc26a85"
Sep 18 16:20:47.178871 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:47.178778    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d039308f8e73fbffbe4577ec730c5c6bb4adec197d9304eadd4292bdc26a85"} err="failed to get container status \"59d039308f8e73fbffbe4577ec730c5c6bb4adec197d9304eadd4292bdc26a85\": rpc error: code = NotFound desc = could not find container \"59d039308f8e73fbffbe4577ec730c5c6bb4adec197d9304eadd4292bdc26a85\": container with ID starting with 59d039308f8e73fbffbe4577ec730c5c6bb4adec197d9304eadd4292bdc26a85 not found: ID does not exist"
Sep 18 16:20:47.178871 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:47.178801    2546 scope.go:117] "RemoveContainer" containerID="33021c135d571a34db677bc9ca404eb2a17819d53c0a5a64fbde7150ef51e0ae"
Sep 18 16:20:47.179101 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:20:47.179083    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33021c135d571a34db677bc9ca404eb2a17819d53c0a5a64fbde7150ef51e0ae\": container with ID starting with 33021c135d571a34db677bc9ca404eb2a17819d53c0a5a64fbde7150ef51e0ae not found: ID does not exist" containerID="33021c135d571a34db677bc9ca404eb2a17819d53c0a5a64fbde7150ef51e0ae"
Sep 18 16:20:47.179157 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:47.179108    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33021c135d571a34db677bc9ca404eb2a17819d53c0a5a64fbde7150ef51e0ae"} err="failed to get container status \"33021c135d571a34db677bc9ca404eb2a17819d53c0a5a64fbde7150ef51e0ae\": rpc error: code = NotFound desc = could not find container \"33021c135d571a34db677bc9ca404eb2a17819d53c0a5a64fbde7150ef51e0ae\": container with ID starting with 33021c135d571a34db677bc9ca404eb2a17819d53c0a5a64fbde7150ef51e0ae not found: ID does not exist"
Sep 18 16:20:47.198562 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:47.198541    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"]
Sep 18 16:20:47.208090 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:47.208070    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-50cecd8f-gw-openshift-default-7854d459cb-4jldx"]
Sep 18 16:20:48.603095 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:48.603066    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d541e34-3351-45d2-a61d-fd3611933989" path="/var/lib/kubelet/pods/5d541e34-3351-45d2-a61d-fd3611933989/volumes"
Sep 18 16:20:50.173296 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.173264    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"]
Sep 18 16:20:50.173747 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.173731    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d541e34-3351-45d2-a61d-fd3611933989" containerName="istio-proxy"
Sep 18 16:20:50.173747 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.173749    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d541e34-3351-45d2-a61d-fd3611933989" containerName="istio-proxy"
Sep 18 16:20:50.173832 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.173771    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d541e34-3351-45d2-a61d-fd3611933989" containerName="istio-proxy"
Sep 18 16:20:50.173832 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.173782    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d541e34-3351-45d2-a61d-fd3611933989" containerName="istio-proxy"
Sep 18 16:20:50.173897 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.173870    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d541e34-3351-45d2-a61d-fd3611933989" containerName="istio-proxy"
Sep 18 16:20:50.173897 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.173887    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d541e34-3351-45d2-a61d-fd3611933989" containerName="istio-proxy"
Sep 18 16:20:50.190165 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.178858    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.190165 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.188217    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1eec1d\"/\"kube-root-ca.crt\""
Sep 18 16:20:50.190165 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.188276    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1eec1d\"/\"openshift-service-ca.crt\""
Sep 18 16:20:50.190165 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.188282    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1eec1d\"/\"simulator-w0-1eec1d-kserve-self-signed-certs\""
Sep 18 16:20:50.190165 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.188785    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1eec1d\"/\"default-dockercfg-wb59m\""
Sep 18 16:20:50.219015 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.218985    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"]
Sep 18 16:20:50.284353 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.284323    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-model-cache\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.284534 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.284392    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-dshm\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.284534 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.284429    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-tmp-dir\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.284658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.284540    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd8kg\" (UniqueName: \"kubernetes.io/projected/97bd1126-9a3e-400f-8350-733958d40f2e-kube-api-access-dd8kg\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.284658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.284622    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-home\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.284754 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.284668    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bd1126-9a3e-400f-8350-733958d40f2e-tls-certs\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.385844 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.385811    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-home\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.386020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.385868    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bd1126-9a3e-400f-8350-733958d40f2e-tls-certs\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.386020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.385929    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-model-cache\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.386020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.385978    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-dshm\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.386020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.385999    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-tmp-dir\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.386283 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.386096    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd8kg\" (UniqueName: \"kubernetes.io/projected/97bd1126-9a3e-400f-8350-733958d40f2e-kube-api-access-dd8kg\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.386283 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.386220    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-home\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.386385 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.386334    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-model-cache\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.386438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.386410    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-tmp-dir\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.388193 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.388172    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-dshm\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.388602 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.388583    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bd1126-9a3e-400f-8350-733958d40f2e-tls-certs\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.402898 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.402874    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd8kg\" (UniqueName: \"kubernetes.io/projected/97bd1126-9a3e-400f-8350-733958d40f2e-kube-api-access-dd8kg\") pod \"simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") " pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.507521 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.507476    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:20:50.822068 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:50.822009    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"]
Sep 18 16:20:50.823965 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:20:50.823939    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97bd1126_9a3e_400f_8350_733958d40f2e.slice/crio-7c91ca44c2ef41292c4f8f6eeba15fe5147ae91c13441730da152d3636557a74 WatchSource:0}: Error finding container 7c91ca44c2ef41292c4f8f6eeba15fe5147ae91c13441730da152d3636557a74: Status 404 returned error can't find the container with id 7c91ca44c2ef41292c4f8f6eeba15fe5147ae91c13441730da152d3636557a74
Sep 18 16:20:51.195094 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:51.195052    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp" event={"ID":"97bd1126-9a3e-400f-8350-733958d40f2e","Type":"ContainerStarted","Data":"964342e43f63a572d951445f1b43a82f1928635b40830eab9760d17b0d6b4f2d"}
Sep 18 16:20:51.195518 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:51.195102    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp" event={"ID":"97bd1126-9a3e-400f-8350-733958d40f2e","Type":"ContainerStarted","Data":"7c91ca44c2ef41292c4f8f6eeba15fe5147ae91c13441730da152d3636557a74"}
Sep 18 16:20:51.246319 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:51.246270    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp" podStartSLOduration=1.246256117 podStartE2EDuration="1.246256117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:51.240549964 +0000 UTC m=+947.149330330" watchObservedRunningTime="2026-09-18 16:20:51.246256117 +0000 UTC m=+947.155036484"
Sep 18 16:20:59.804354 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:59.804315    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"]
Sep 18 16:20:59.807349 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:59.807326    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:20:59.814899 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:59.814874    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-eef18d\"/\"openshift-service-ca.crt\""
Sep 18 16:20:59.815065 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:59.815048    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-eef18d\"/\"kube-root-ca.crt\""
Sep 18 16:20:59.815168 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:20:59.815130    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"autores-model-eef18d-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-autores-eef18d\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-autores-eef18d\"/\"autores-model-eef18d-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 18 16:20:59.815168 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:20:59.815144    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"autores-model-eef18d-kserve-69b45d847-l76nk\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-autores-eef18d\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="4adfef27-ceca-4f3d-9f21-07f76551f803" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:20:59.815287 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:20:59.815226    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-chdb7\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"ai-tenant-e2e-autores-eef18d\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"ai-tenant-e2e-autores-eef18d\"/\"default-dockercfg-chdb7\"" type="*v1.Secret"
Sep 18 16:20:59.838732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:59.838707    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"]
Sep 18 16:20:59.891749 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:59.891715    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-dshm\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:20:59.901056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:59.892044    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4adfef27-ceca-4f3d-9f21-07f76551f803-tls-certs\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:20:59.901292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:59.901240    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-home\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:20:59.901465 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:59.901427    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-model-cache\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:20:59.901606 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:59.901590    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgfvt\" (UniqueName: \"kubernetes.io/projected/4adfef27-ceca-4f3d-9f21-07f76551f803-kube-api-access-dgfvt\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:20:59.901722 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:20:59.901709    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-tmp-dir\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:00.003418 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.003374    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-model-cache\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:00.003624 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.003446    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgfvt\" (UniqueName: \"kubernetes.io/projected/4adfef27-ceca-4f3d-9f21-07f76551f803-kube-api-access-dgfvt\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:00.003624 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.003479    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-tmp-dir\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:00.003624 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.003527    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-dshm\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:00.003799 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.003654    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4adfef27-ceca-4f3d-9f21-07f76551f803-tls-certs\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:00.003799 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.003683    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-home\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:00.004142 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.004117    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-home\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:00.004434 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.004409    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-tmp-dir\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:00.004733 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.004711    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-model-cache\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:00.006514 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.006495    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-dshm\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:00.017871 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.017850    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgfvt\" (UniqueName: \"kubernetes.io/projected/4adfef27-ceca-4f3d-9f21-07f76551f803-kube-api-access-dgfvt\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:00.508454 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.508416    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:21:00.508610 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.508470    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:21:00.520146 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.520121    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:21:00.668776 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.668748    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-eef18d\"/\"autores-model-eef18d-kserve-self-signed-certs\""
Sep 18 16:21:00.677418 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.677391    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4adfef27-ceca-4f3d-9f21-07f76551f803-tls-certs\") pod \"autores-model-eef18d-kserve-69b45d847-l76nk\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") " pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:00.892367 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.892111    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-eef18d\"/\"default-dockercfg-chdb7\""
Sep 18 16:21:00.899047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:00.898811    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:01.188268 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:01.188178    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"]
Sep 18 16:21:01.206617 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:21:01.206583    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4adfef27_ceca_4f3d_9f21_07f76551f803.slice/crio-f17aa4c74f8b1d7c173bd163adeeeb7330868e3ea6eaef4632ee0bf5c4579e8a WatchSource:0}: Error finding container f17aa4c74f8b1d7c173bd163adeeeb7330868e3ea6eaef4632ee0bf5c4579e8a: Status 404 returned error can't find the container with id f17aa4c74f8b1d7c173bd163adeeeb7330868e3ea6eaef4632ee0bf5c4579e8a
Sep 18 16:21:01.251904 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:01.251876    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk" event={"ID":"4adfef27-ceca-4f3d-9f21-07f76551f803","Type":"ContainerStarted","Data":"f17aa4c74f8b1d7c173bd163adeeeb7330868e3ea6eaef4632ee0bf5c4579e8a"}
Sep 18 16:21:01.263671 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:01.263617    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:21:02.258189 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:02.258148    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk" event={"ID":"4adfef27-ceca-4f3d-9f21-07f76551f803","Type":"ContainerStarted","Data":"30f4f93563f7440e22b4fb58ab6379dd65072c0b826ffa78eab0ff4e656172e0"}
Sep 18 16:21:07.299945 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:07.293786    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" event={"ID":"f827731d-db49-4d99-b566-900a178d7052","Type":"ContainerDied","Data":"649df7c629f6024ec87bad26b51ff82a28c952ab381ba5296932cc4a55234d7c"}
Sep 18 16:21:07.299945 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:07.293914    2546 generic.go:356] "Generic (PLEG): container finished" podID="f827731d-db49-4d99-b566-900a178d7052" containerID="649df7c629f6024ec87bad26b51ff82a28c952ab381ba5296932cc4a55234d7c" exitCode=0
Sep 18 16:21:07.299945 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:07.294458    2546 scope.go:117] "RemoveContainer" containerID="649df7c629f6024ec87bad26b51ff82a28c952ab381ba5296932cc4a55234d7c"
Sep 18 16:21:07.300863 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:07.300810    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:21:07.300919 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:07.300906    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:21:07.300957 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:07.300943    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:21:07.320692 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:07.320545    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk" podStartSLOduration=8.320529898 podStartE2EDuration="8.320529898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:21:02.290953043 +0000 UTC m=+958.199733411" watchObservedRunningTime="2026-09-18 16:21:07.320529898 +0000 UTC m=+963.229310252"
Sep 18 16:21:08.303414 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:08.303383    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" event={"ID":"f827731d-db49-4d99-b566-900a178d7052","Type":"ContainerStarted","Data":"c1cd822aba11ae1916903b6140d9f8c740abfe398b11506b63ba66ecb0582f4b"}
Sep 18 16:21:09.016768 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:09.016726    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:21:09.017967 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:09.017937    2546 patch_prober.go:28] interesting pod/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 18 16:21:09.018119 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:09.017998    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" podUID="f827731d-db49-4d99-b566-900a178d7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 16:21:10.016359 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:10.016316    2546 patch_prober.go:28] interesting pod/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 18 16:21:10.016814 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:10.016382    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" podUID="f827731d-db49-4d99-b566-900a178d7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 16:21:10.162311 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:10.162270    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:21:10.902517 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:10.901348    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:10.902517 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:10.901395    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:10.917958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:10.917931    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:11.016624 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:11.016586    2546 patch_prober.go:28] interesting pod/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 18 16:21:11.017091 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:11.016642    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" podUID="f827731d-db49-4d99-b566-900a178d7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 16:21:11.337103 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:11.337068    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:21:12.016209 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:12.016174    2546 patch_prober.go:28] interesting pod/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 18 16:21:12.016455 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:12.016426    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" podUID="f827731d-db49-4d99-b566-900a178d7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 16:21:13.016817 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:13.016781    2546 patch_prober.go:28] interesting pod/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 18 16:21:13.017218 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:13.016847    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" podUID="f827731d-db49-4d99-b566-900a178d7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 16:21:13.140272 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:13.140235    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:21:14.016364 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:14.016319    2546 patch_prober.go:28] interesting pod/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Sep 18 16:21:14.016593 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:14.016378    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" podUID="f827731d-db49-4d99-b566-900a178d7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Sep 18 16:21:15.021682 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:15.021650    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:21:15.022511 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:15.022493    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:21:19.635149 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:19.635119    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r"]
Sep 18 16:21:19.637977 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:19.637956    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r"
Sep 18 16:21:19.640982 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:19.640958    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:21:19.641096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:19.641059    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:21:19.652720 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:19.652698    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r"]
Sep 18 16:21:19.712266 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:19.712237    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f0b0bb-aea2-49ed-95ea-e167d5d6995f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-48de7216-bzc5r\" (UID: \"75f0b0bb-aea2-49ed-95ea-e167d5d6995f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r"
Sep 18 16:21:19.813544 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:19.813516    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f0b0bb-aea2-49ed-95ea-e167d5d6995f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-48de7216-bzc5r\" (UID: \"75f0b0bb-aea2-49ed-95ea-e167d5d6995f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r"
Sep 18 16:21:19.814150 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:19.814131    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f0b0bb-aea2-49ed-95ea-e167d5d6995f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-48de7216-bzc5r\" (UID: \"75f0b0bb-aea2-49ed-95ea-e167d5d6995f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r"
Sep 18 16:21:19.947408 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:19.947379    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r"
Sep 18 16:21:20.243108 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:20.243079    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r"]
Sep 18 16:21:20.245649 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:21:20.245621    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f0b0bb_aea2_49ed_95ea_e167d5d6995f.slice/crio-8b1832229da5b458eeac93e489e4c8cf1eec69cda25ee194b78d4b292b43514a WatchSource:0}: Error finding container 8b1832229da5b458eeac93e489e4c8cf1eec69cda25ee194b78d4b292b43514a: Status 404 returned error can't find the container with id 8b1832229da5b458eeac93e489e4c8cf1eec69cda25ee194b78d4b292b43514a
Sep 18 16:21:20.372575 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:20.372547    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r" event={"ID":"75f0b0bb-aea2-49ed-95ea-e167d5d6995f","Type":"ContainerStarted","Data":"8b1832229da5b458eeac93e489e4c8cf1eec69cda25ee194b78d4b292b43514a"}
Sep 18 16:21:21.380145 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:21.380066    2546 generic.go:356] "Generic (PLEG): container finished" podID="75f0b0bb-aea2-49ed-95ea-e167d5d6995f" containerID="ebfe849a043aec4aa3c61ab4e654e6eb39b0ba6190cf74331d94326c187adc23" exitCode=0
Sep 18 16:21:21.380145 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:21.380128    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r" event={"ID":"75f0b0bb-aea2-49ed-95ea-e167d5d6995f","Type":"ContainerDied","Data":"ebfe849a043aec4aa3c61ab4e654e6eb39b0ba6190cf74331d94326c187adc23"}
Sep 18 16:21:22.529565 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:22.529542    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r"
Sep 18 16:21:22.644354 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:22.644280    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f0b0bb-aea2-49ed-95ea-e167d5d6995f-maas-api-service-ca\") pod \"75f0b0bb-aea2-49ed-95ea-e167d5d6995f\" (UID: \"75f0b0bb-aea2-49ed-95ea-e167d5d6995f\") "
Sep 18 16:21:22.644597 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:22.644574    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f0b0bb-aea2-49ed-95ea-e167d5d6995f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "75f0b0bb-aea2-49ed-95ea-e167d5d6995f" (UID: "75f0b0bb-aea2-49ed-95ea-e167d5d6995f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:21:22.745499 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:22.745464    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75f0b0bb-aea2-49ed-95ea-e167d5d6995f-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:21:23.392406 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:23.392371    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r"
Sep 18 16:21:23.392406 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:23.392389    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r" event={"ID":"75f0b0bb-aea2-49ed-95ea-e167d5d6995f","Type":"ContainerDied","Data":"8b1832229da5b458eeac93e489e4c8cf1eec69cda25ee194b78d4b292b43514a"}
Sep 18 16:21:23.392610 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:23.392445    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b1832229da5b458eeac93e489e4c8cf1eec69cda25ee194b78d4b292b43514a"
Sep 18 16:21:33.191777 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:33.191682    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h"]
Sep 18 16:21:33.192369 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:33.192284    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75f0b0bb-aea2-49ed-95ea-e167d5d6995f" containerName="revoke-keys"
Sep 18 16:21:33.192369 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:33.192305    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="75f0b0bb-aea2-49ed-95ea-e167d5d6995f" containerName="revoke-keys"
Sep 18 16:21:33.192483 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:33.192452    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="75f0b0bb-aea2-49ed-95ea-e167d5d6995f" containerName="revoke-keys"
Sep 18 16:21:33.194515 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:33.194495    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h"
Sep 18 16:21:33.205098 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:33.205064    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:21:33.205234 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:33.205096    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:21:33.222452 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:33.222426    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h"]
Sep 18 16:21:33.250394 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:33.250362    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107f42e1-5ebf-40a2-b547-049a16a386bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-eef18d-jds4h\" (UID: \"107f42e1-5ebf-40a2-b547-049a16a386bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h"
Sep 18 16:21:33.351852 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:33.351816    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107f42e1-5ebf-40a2-b547-049a16a386bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-eef18d-jds4h\" (UID: \"107f42e1-5ebf-40a2-b547-049a16a386bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h"
Sep 18 16:21:33.352412 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:33.352393    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107f42e1-5ebf-40a2-b547-049a16a386bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-eef18d-jds4h\" (UID: \"107f42e1-5ebf-40a2-b547-049a16a386bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h"
Sep 18 16:21:33.510968 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:33.510940    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h"
Sep 18 16:21:33.789580 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:33.789554    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h"]
Sep 18 16:21:33.790495 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:21:33.790469    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107f42e1_5ebf_40a2_b547_049a16a386bd.slice/crio-b27c5b634267a5be659beee8bdca5ea127ae9b3c673d6e574e98694e6442cedd WatchSource:0}: Error finding container b27c5b634267a5be659beee8bdca5ea127ae9b3c673d6e574e98694e6442cedd: Status 404 returned error can't find the container with id b27c5b634267a5be659beee8bdca5ea127ae9b3c673d6e574e98694e6442cedd
Sep 18 16:21:34.451565 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:34.451535    2546 generic.go:356] "Generic (PLEG): container finished" podID="107f42e1-5ebf-40a2-b547-049a16a386bd" containerID="02f40b0c4012900cc3ecdda64ec3731ec516933cb1e96180836fbe49f20684ed" exitCode=0
Sep 18 16:21:34.451897 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:34.451603    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h" event={"ID":"107f42e1-5ebf-40a2-b547-049a16a386bd","Type":"ContainerDied","Data":"02f40b0c4012900cc3ecdda64ec3731ec516933cb1e96180836fbe49f20684ed"}
Sep 18 16:21:34.451897 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:34.451650    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h" event={"ID":"107f42e1-5ebf-40a2-b547-049a16a386bd","Type":"ContainerStarted","Data":"b27c5b634267a5be659beee8bdca5ea127ae9b3c673d6e574e98694e6442cedd"}
Sep 18 16:21:35.315723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:35.315685    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r"]
Sep 18 16:21:35.320190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:35.320157    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-48de7216-bzc5r"]
Sep 18 16:21:35.616691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:35.616670    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h"
Sep 18 16:21:35.676119 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:35.676094    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107f42e1-5ebf-40a2-b547-049a16a386bd-maas-api-service-ca\") pod \"107f42e1-5ebf-40a2-b547-049a16a386bd\" (UID: \"107f42e1-5ebf-40a2-b547-049a16a386bd\") "
Sep 18 16:21:35.676415 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:35.676394    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107f42e1-5ebf-40a2-b547-049a16a386bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "107f42e1-5ebf-40a2-b547-049a16a386bd" (UID: "107f42e1-5ebf-40a2-b547-049a16a386bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:21:35.676574 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:35.676558    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/107f42e1-5ebf-40a2-b547-049a16a386bd-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:21:36.466925 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:36.466896    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h"
Sep 18 16:21:36.467130 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:36.466928    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h" event={"ID":"107f42e1-5ebf-40a2-b547-049a16a386bd","Type":"ContainerDied","Data":"b27c5b634267a5be659beee8bdca5ea127ae9b3c673d6e574e98694e6442cedd"}
Sep 18 16:21:36.467130 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:36.466986    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b27c5b634267a5be659beee8bdca5ea127ae9b3c673d6e574e98694e6442cedd"
Sep 18 16:21:36.485159 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:36.485131    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"]
Sep 18 16:21:36.485320 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:36.485300    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" podUID="f827731d-db49-4d99-b566-900a178d7052" containerName="istio-proxy" containerID="cri-o://c1cd822aba11ae1916903b6140d9f8c740abfe398b11506b63ba66ecb0582f4b" gracePeriod=30
Sep 18 16:21:36.607752 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:36.607331    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75f0b0bb-aea2-49ed-95ea-e167d5d6995f" path="/var/lib/kubelet/pods/75f0b0bb-aea2-49ed-95ea-e167d5d6995f/volumes"
Sep 18 16:21:37.221251 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.221220    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"]
Sep 18 16:21:37.221767 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.221739    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="107f42e1-5ebf-40a2-b547-049a16a386bd" containerName="revoke-keys"
Sep 18 16:21:37.221767 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.221759    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="107f42e1-5ebf-40a2-b547-049a16a386bd" containerName="revoke-keys"
Sep 18 16:21:37.221848 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.221828    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="107f42e1-5ebf-40a2-b547-049a16a386bd" containerName="revoke-keys"
Sep 18 16:21:37.224151 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.224130    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.227778 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.227756    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-11860c1b-openshift-default-dockercfg-fvqz6\""
Sep 18 16:21:37.242354 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.242329    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"]
Sep 18 16:21:37.293139 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.293110    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g5zr\" (UniqueName: \"kubernetes.io/projected/a2036874-8eb8-4d66-bf00-b0d140e9a197-kube-api-access-5g5zr\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.293291 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.293157    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2036874-8eb8-4d66-bf00-b0d140e9a197-istiod-ca-cert\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.293291 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.293185    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-workload-certs\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.293414 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.293302    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-workload-socket\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.293414 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.293339    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-credential-socket\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.293414 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.293364    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-podinfo\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.293414 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.293396    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-token\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.293627 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.293440    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-envoy\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.293627 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.293456    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-data\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.394264 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.394231    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g5zr\" (UniqueName: \"kubernetes.io/projected/a2036874-8eb8-4d66-bf00-b0d140e9a197-kube-api-access-5g5zr\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.394434 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.394276    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2036874-8eb8-4d66-bf00-b0d140e9a197-istiod-ca-cert\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.394434 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.394320    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-workload-certs\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.394434 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.394395    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-workload-socket\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.394434 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.394428    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-credential-socket\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.394603 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.394473    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-podinfo\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.394603 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.394511    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-token\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.394603 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.394569    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-envoy\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.394603 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.394595    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-data\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.394864 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.394836    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-workload-socket\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.394864 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.394849    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-workload-certs\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.394993 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.394911    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-credential-socket\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.394993 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.394965    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-data\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.395352 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.395336    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2036874-8eb8-4d66-bf00-b0d140e9a197-istiod-ca-cert\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.396778 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.396757    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-envoy\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.396940 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.396920    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-podinfo\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.410817 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.410791    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-token\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.411579 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.411558    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g5zr\" (UniqueName: \"kubernetes.io/projected/a2036874-8eb8-4d66-bf00-b0d140e9a197-kube-api-access-5g5zr\") pod \"e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") " pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.535768 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.535687    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:37.819244 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.819166    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"]
Sep 18 16:21:37.823265 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:21:37.822490    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2036874_8eb8_4d66_bf00_b0d140e9a197.slice/crio-ed75b5de95c8e0b947b9ac7fc6edc6cf9f6709603ec6b079062df42de3898858 WatchSource:0}: Error finding container ed75b5de95c8e0b947b9ac7fc6edc6cf9f6709603ec6b079062df42de3898858: Status 404 returned error can't find the container with id ed75b5de95c8e0b947b9ac7fc6edc6cf9f6709603ec6b079062df42de3898858
Sep 18 16:21:37.825604 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.825566    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:21:37.825696 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.825637    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:21:37.825696 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:37.825669    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:21:38.480308 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:38.480268    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj" event={"ID":"a2036874-8eb8-4d66-bf00-b0d140e9a197","Type":"ContainerStarted","Data":"4e72a64fbfe72a9d8584c68849712a2017613d6c2765fec369680a4955d9e71c"}
Sep 18 16:21:38.480754 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:38.480333    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj" event={"ID":"a2036874-8eb8-4d66-bf00-b0d140e9a197","Type":"ContainerStarted","Data":"ed75b5de95c8e0b947b9ac7fc6edc6cf9f6709603ec6b079062df42de3898858"}
Sep 18 16:21:38.536078 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:38.536049    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:38.540877 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:38.540857    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:38.574706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:38.574657    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj" podStartSLOduration=1.574644586 podStartE2EDuration="1.574644586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:21:38.518923298 +0000 UTC m=+994.427703648" watchObservedRunningTime="2026-09-18 16:21:38.574644586 +0000 UTC m=+994.483424940"
Sep 18 16:21:39.486830 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:39.486796    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:39.487734 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:39.487717    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:21:41.728381 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.728357    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:21:41.843214 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.843133    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-workload-socket\") pod \"f827731d-db49-4d99-b566-900a178d7052\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") "
Sep 18 16:21:41.843214 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.843169    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-credential-socket\") pod \"f827731d-db49-4d99-b566-900a178d7052\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") "
Sep 18 16:21:41.843214 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.843187    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f827731d-db49-4d99-b566-900a178d7052-istio-podinfo\") pod \"f827731d-db49-4d99-b566-900a178d7052\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") "
Sep 18 16:21:41.843214 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.843217    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f827731d-db49-4d99-b566-900a178d7052-istio-token\") pod \"f827731d-db49-4d99-b566-900a178d7052\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") "
Sep 18 16:21:41.843489 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.843254    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-istio-envoy\") pod \"f827731d-db49-4d99-b566-900a178d7052\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") "
Sep 18 16:21:41.843489 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.843327    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-workload-certs\") pod \"f827731d-db49-4d99-b566-900a178d7052\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") "
Sep 18 16:21:41.843489 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.843473    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-istio-data\") pod \"f827731d-db49-4d99-b566-900a178d7052\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") "
Sep 18 16:21:41.843646 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.843452    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f827731d-db49-4d99-b566-900a178d7052" (UID: "f827731d-db49-4d99-b566-900a178d7052"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:21:41.843646 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.843518    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f827731d-db49-4d99-b566-900a178d7052" (UID: "f827731d-db49-4d99-b566-900a178d7052"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:21:41.843646 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.843517    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px5bw\" (UniqueName: \"kubernetes.io/projected/f827731d-db49-4d99-b566-900a178d7052-kube-api-access-px5bw\") pod \"f827731d-db49-4d99-b566-900a178d7052\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") "
Sep 18 16:21:41.843646 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.843611    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f827731d-db49-4d99-b566-900a178d7052-istiod-ca-cert\") pod \"f827731d-db49-4d99-b566-900a178d7052\" (UID: \"f827731d-db49-4d99-b566-900a178d7052\") "
Sep 18 16:21:41.843830 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.843675    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-istio-data" (OuterVolumeSpecName: "istio-data") pod "f827731d-db49-4d99-b566-900a178d7052" (UID: "f827731d-db49-4d99-b566-900a178d7052"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:21:41.844049 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.844004    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f827731d-db49-4d99-b566-900a178d7052-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f827731d-db49-4d99-b566-900a178d7052" (UID: "f827731d-db49-4d99-b566-900a178d7052"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:21:41.844316 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.844158    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-istio-data\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:21:41.844316 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.844323    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-workload-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:21:41.844502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.844334    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-credential-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:21:41.844502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.844149    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f827731d-db49-4d99-b566-900a178d7052" (UID: "f827731d-db49-4d99-b566-900a178d7052"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:21:41.845621 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.845598    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f827731d-db49-4d99-b566-900a178d7052-kube-api-access-px5bw" (OuterVolumeSpecName: "kube-api-access-px5bw") pod "f827731d-db49-4d99-b566-900a178d7052" (UID: "f827731d-db49-4d99-b566-900a178d7052"). InnerVolumeSpecName "kube-api-access-px5bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:21:41.845707 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.845660    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f827731d-db49-4d99-b566-900a178d7052-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f827731d-db49-4d99-b566-900a178d7052" (UID: "f827731d-db49-4d99-b566-900a178d7052"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:21:41.845886 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.845871    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f827731d-db49-4d99-b566-900a178d7052-istio-token" (OuterVolumeSpecName: "istio-token") pod "f827731d-db49-4d99-b566-900a178d7052" (UID: "f827731d-db49-4d99-b566-900a178d7052"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:21:41.845938 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.845888    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f827731d-db49-4d99-b566-900a178d7052" (UID: "f827731d-db49-4d99-b566-900a178d7052"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:21:41.944889 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.944858    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-workload-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:21:41.944889 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.944887    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px5bw\" (UniqueName: \"kubernetes.io/projected/f827731d-db49-4d99-b566-900a178d7052-kube-api-access-px5bw\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:21:41.944889 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.944898    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f827731d-db49-4d99-b566-900a178d7052-istiod-ca-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:21:41.945145 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.944907    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f827731d-db49-4d99-b566-900a178d7052-istio-podinfo\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:21:41.945145 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.944915    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f827731d-db49-4d99-b566-900a178d7052-istio-token\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:21:41.945145 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:41.944924    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f827731d-db49-4d99-b566-900a178d7052-istio-envoy\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:21:42.504199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.504162    2546 generic.go:356] "Generic (PLEG): container finished" podID="f827731d-db49-4d99-b566-900a178d7052" containerID="c1cd822aba11ae1916903b6140d9f8c740abfe398b11506b63ba66ecb0582f4b" exitCode=0
Sep 18 16:21:42.504362 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.504213    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"
Sep 18 16:21:42.504362 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.504244    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" event={"ID":"f827731d-db49-4d99-b566-900a178d7052","Type":"ContainerDied","Data":"c1cd822aba11ae1916903b6140d9f8c740abfe398b11506b63ba66ecb0582f4b"}
Sep 18 16:21:42.504362 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.504323    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj" event={"ID":"f827731d-db49-4d99-b566-900a178d7052","Type":"ContainerDied","Data":"96a848be9be7e9d99a16edb7744295ffbfba9816a5e72eda0a7ad1ac762c0fba"}
Sep 18 16:21:42.504362 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.504344    2546 scope.go:117] "RemoveContainer" containerID="c1cd822aba11ae1916903b6140d9f8c740abfe398b11506b63ba66ecb0582f4b"
Sep 18 16:21:42.514370 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.514352    2546 scope.go:117] "RemoveContainer" containerID="649df7c629f6024ec87bad26b51ff82a28c952ab381ba5296932cc4a55234d7c"
Sep 18 16:21:42.522474 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.522455    2546 scope.go:117] "RemoveContainer" containerID="c1cd822aba11ae1916903b6140d9f8c740abfe398b11506b63ba66ecb0582f4b"
Sep 18 16:21:42.522736 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:21:42.522716    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1cd822aba11ae1916903b6140d9f8c740abfe398b11506b63ba66ecb0582f4b\": container with ID starting with c1cd822aba11ae1916903b6140d9f8c740abfe398b11506b63ba66ecb0582f4b not found: ID does not exist" containerID="c1cd822aba11ae1916903b6140d9f8c740abfe398b11506b63ba66ecb0582f4b"
Sep 18 16:21:42.522805 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.522749    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1cd822aba11ae1916903b6140d9f8c740abfe398b11506b63ba66ecb0582f4b"} err="failed to get container status \"c1cd822aba11ae1916903b6140d9f8c740abfe398b11506b63ba66ecb0582f4b\": rpc error: code = NotFound desc = could not find container \"c1cd822aba11ae1916903b6140d9f8c740abfe398b11506b63ba66ecb0582f4b\": container with ID starting with c1cd822aba11ae1916903b6140d9f8c740abfe398b11506b63ba66ecb0582f4b not found: ID does not exist"
Sep 18 16:21:42.522805 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.522776    2546 scope.go:117] "RemoveContainer" containerID="649df7c629f6024ec87bad26b51ff82a28c952ab381ba5296932cc4a55234d7c"
Sep 18 16:21:42.523047 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:21:42.523014    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649df7c629f6024ec87bad26b51ff82a28c952ab381ba5296932cc4a55234d7c\": container with ID starting with 649df7c629f6024ec87bad26b51ff82a28c952ab381ba5296932cc4a55234d7c not found: ID does not exist" containerID="649df7c629f6024ec87bad26b51ff82a28c952ab381ba5296932cc4a55234d7c"
Sep 18 16:21:42.523084 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.523053    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649df7c629f6024ec87bad26b51ff82a28c952ab381ba5296932cc4a55234d7c"} err="failed to get container status \"649df7c629f6024ec87bad26b51ff82a28c952ab381ba5296932cc4a55234d7c\": rpc error: code = NotFound desc = could not find container \"649df7c629f6024ec87bad26b51ff82a28c952ab381ba5296932cc4a55234d7c\": container with ID starting with 649df7c629f6024ec87bad26b51ff82a28c952ab381ba5296932cc4a55234d7c not found: ID does not exist"
Sep 18 16:21:42.552913 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.552883    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"]
Sep 18 16:21:42.562071 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.562046    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-48de7216-openshift-default-848c856ffd-mkkfj"]
Sep 18 16:21:42.603621 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.603585    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f827731d-db49-4d99-b566-900a178d7052" path="/var/lib/kubelet/pods/f827731d-db49-4d99-b566-900a178d7052/volumes"
Sep 18 16:21:42.694939 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:42.694908    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:21:48.580707 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.580663    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"]
Sep 18 16:21:48.581259 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.581240    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f827731d-db49-4d99-b566-900a178d7052" containerName="istio-proxy"
Sep 18 16:21:48.581326 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.581261    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f827731d-db49-4d99-b566-900a178d7052" containerName="istio-proxy"
Sep 18 16:21:48.581391 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.581379    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f827731d-db49-4d99-b566-900a178d7052" containerName="istio-proxy"
Sep 18 16:21:48.581441 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.581390    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f827731d-db49-4d99-b566-900a178d7052" containerName="istio-proxy"
Sep 18 16:21:48.584667 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.584646    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.590215 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.590190    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fbc529\"/\"default-dockercfg-g8hb4\""
Sep 18 16:21:48.590334 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.590233    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fbc529\"/\"openshift-service-ca.crt\""
Sep 18 16:21:48.591372 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.591349    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fbc529\"/\"kube-root-ca.crt\""
Sep 18 16:21:48.591484 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.591390    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fbc529\"/\"test-model-fbc529-kserve-self-signed-certs\""
Sep 18 16:21:48.612612 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.612591    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"]
Sep 18 16:21:48.719575 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.719545    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-dshm\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.719694 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.719581    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-model-cache\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.719694 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.719633    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-home\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.719694 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.719665    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-tmp-dir\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.719832 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.719693    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309ae6b9-331c-46c1-9936-9e22dfa20dd9-tls-certs\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.719832 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.719812    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfgdw\" (UniqueName: \"kubernetes.io/projected/309ae6b9-331c-46c1-9936-9e22dfa20dd9-kube-api-access-qfgdw\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.820850 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.820807    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfgdw\" (UniqueName: \"kubernetes.io/projected/309ae6b9-331c-46c1-9936-9e22dfa20dd9-kube-api-access-qfgdw\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.821020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.820875    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-dshm\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.821020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.820894    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-model-cache\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.821020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.820925    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-home\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.821020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.820948    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-tmp-dir\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.821020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.820973    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309ae6b9-331c-46c1-9936-9e22dfa20dd9-tls-certs\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.821417 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.821397    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-home\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.821479 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.821414    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-tmp-dir\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.821479 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.821449    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-model-cache\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.823145 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.823126    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-dshm\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.823310 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.823293    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309ae6b9-331c-46c1-9936-9e22dfa20dd9-tls-certs\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.832505 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.832452    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfgdw\" (UniqueName: \"kubernetes.io/projected/309ae6b9-331c-46c1-9936-9e22dfa20dd9-kube-api-access-qfgdw\") pod \"test-model-fbc529-kserve-59579bccbc-5rdns\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") " pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:48.894249 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:48.894176    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:49.187686 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:49.187662    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"]
Sep 18 16:21:49.189610 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:21:49.189581    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod309ae6b9_331c_46c1_9936_9e22dfa20dd9.slice/crio-ace0fd3c3de3735d75613b3b10276b190834ed048fb89e9d9e28ba15da066aea WatchSource:0}: Error finding container ace0fd3c3de3735d75613b3b10276b190834ed048fb89e9d9e28ba15da066aea: Status 404 returned error can't find the container with id ace0fd3c3de3735d75613b3b10276b190834ed048fb89e9d9e28ba15da066aea
Sep 18 16:21:49.545795 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:49.545763    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns" event={"ID":"309ae6b9-331c-46c1-9936-9e22dfa20dd9","Type":"ContainerStarted","Data":"ff53d9e51b884738447cd73644f3306d0fd0cd09d26b816b80a404ac9acf3316"}
Sep 18 16:21:49.546005 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:49.545803    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns" event={"ID":"309ae6b9-331c-46c1-9936-9e22dfa20dd9","Type":"ContainerStarted","Data":"ace0fd3c3de3735d75613b3b10276b190834ed048fb89e9d9e28ba15da066aea"}
Sep 18 16:21:49.580514 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:49.580465    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns" podStartSLOduration=1.580449909 podStartE2EDuration="1.580449909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:21:49.579328021 +0000 UTC m=+1005.488108373" watchObservedRunningTime="2026-09-18 16:21:49.580449909 +0000 UTC m=+1005.489230260"
Sep 18 16:21:51.098813 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:51.098770    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h"]
Sep 18 16:21:51.114055 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:51.108569    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-eef18d-jds4h"]
Sep 18 16:21:52.604873 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:52.604842    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="107f42e1-5ebf-40a2-b547-049a16a386bd" path="/var/lib/kubelet/pods/107f42e1-5ebf-40a2-b547-049a16a386bd/volumes"
Sep 18 16:21:56.465744 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:56.465706    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"]
Sep 18 16:21:58.894529 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:58.894495    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:58.894880 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:58.894548    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:58.906326 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:58.906302    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:59.615518 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:59.615489    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:21:59.970764 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:21:59.970726    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:22:04.800079 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:04.800039    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7"]
Sep 18 16:22:04.800640 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:04.800621    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f827731d-db49-4d99-b566-900a178d7052" containerName="istio-proxy"
Sep 18 16:22:04.800741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:04.800642    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f827731d-db49-4d99-b566-900a178d7052" containerName="istio-proxy"
Sep 18 16:22:04.803736 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:04.803712    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7"
Sep 18 16:22:04.807188 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:04.807164    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:22:04.807401 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:04.807377    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:22:04.824365 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:04.824343    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7"]
Sep 18 16:22:04.898513 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:04.898473    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9f6c0c-952f-49e1-81a0-ff0a791134ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7\" (UID: \"0b9f6c0c-952f-49e1-81a0-ff0a791134ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7"
Sep 18 16:22:05.000633 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:05.000531    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9f6c0c-952f-49e1-81a0-ff0a791134ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7\" (UID: \"0b9f6c0c-952f-49e1-81a0-ff0a791134ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7"
Sep 18 16:22:05.002574 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:05.001763    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9f6c0c-952f-49e1-81a0-ff0a791134ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7\" (UID: \"0b9f6c0c-952f-49e1-81a0-ff0a791134ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7"
Sep 18 16:22:05.116047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:05.115995    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7"
Sep 18 16:22:05.418225 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:05.418195    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7"]
Sep 18 16:22:05.419070 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:22:05.419043    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9f6c0c_952f_49e1_81a0_ff0a791134ec.slice/crio-1bebd26590acbeb767c8b1d04732d71ae608c6ba238e7cb2fda1e11199c5946c WatchSource:0}: Error finding container 1bebd26590acbeb767c8b1d04732d71ae608c6ba238e7cb2fda1e11199c5946c: Status 404 returned error can't find the container with id 1bebd26590acbeb767c8b1d04732d71ae608c6ba238e7cb2fda1e11199c5946c
Sep 18 16:22:05.642931 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:05.642887    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7" event={"ID":"0b9f6c0c-952f-49e1-81a0-ff0a791134ec","Type":"ContainerStarted","Data":"1bebd26590acbeb767c8b1d04732d71ae608c6ba238e7cb2fda1e11199c5946c"}
Sep 18 16:22:06.650343 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:06.650245    2546 generic.go:356] "Generic (PLEG): container finished" podID="0b9f6c0c-952f-49e1-81a0-ff0a791134ec" containerID="6a4655cdfe131ff9657eff7d31f3b57f918c268eb3fe094530bb0a00a874dc11" exitCode=0
Sep 18 16:22:06.650343 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:06.650292    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7" event={"ID":"0b9f6c0c-952f-49e1-81a0-ff0a791134ec","Type":"ContainerDied","Data":"6a4655cdfe131ff9657eff7d31f3b57f918c268eb3fe094530bb0a00a874dc11"}
Sep 18 16:22:07.793328 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:07.793297    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7"
Sep 18 16:22:07.833476 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:07.833438    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9f6c0c-952f-49e1-81a0-ff0a791134ec-maas-api-service-ca\") pod \"0b9f6c0c-952f-49e1-81a0-ff0a791134ec\" (UID: \"0b9f6c0c-952f-49e1-81a0-ff0a791134ec\") "
Sep 18 16:22:07.833809 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:07.833785    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b9f6c0c-952f-49e1-81a0-ff0a791134ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b9f6c0c-952f-49e1-81a0-ff0a791134ec" (UID: "0b9f6c0c-952f-49e1-81a0-ff0a791134ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:22:07.833999 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:07.833979    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b9f6c0c-952f-49e1-81a0-ff0a791134ec-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:08.661852 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:08.661823    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7"
Sep 18 16:22:08.662088 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:08.661850    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7" event={"ID":"0b9f6c0c-952f-49e1-81a0-ff0a791134ec","Type":"ContainerDied","Data":"1bebd26590acbeb767c8b1d04732d71ae608c6ba238e7cb2fda1e11199c5946c"}
Sep 18 16:22:08.662088 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:08.661899    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bebd26590acbeb767c8b1d04732d71ae608c6ba238e7cb2fda1e11199c5946c"
Sep 18 16:22:08.663645 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:08.663621    2546 generic.go:356] "Generic (PLEG): container finished" podID="a2036874-8eb8-4d66-bf00-b0d140e9a197" containerID="4e72a64fbfe72a9d8584c68849712a2017613d6c2765fec369680a4955d9e71c" exitCode=0
Sep 18 16:22:08.663770 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:08.663663    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj" event={"ID":"a2036874-8eb8-4d66-bf00-b0d140e9a197","Type":"ContainerDied","Data":"4e72a64fbfe72a9d8584c68849712a2017613d6c2765fec369680a4955d9e71c"}
Sep 18 16:22:08.664042 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:08.664015    2546 scope.go:117] "RemoveContainer" containerID="4e72a64fbfe72a9d8584c68849712a2017613d6c2765fec369680a4955d9e71c"
Sep 18 16:22:08.665120 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:08.665087    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:22:08.665180 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:08.665158    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:22:08.665216 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:08.665187    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:22:09.487564 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:09.487531    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:22:09.670821 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:09.670791    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj" event={"ID":"a2036874-8eb8-4d66-bf00-b0d140e9a197","Type":"ContainerStarted","Data":"d0f1fdbddd6cee452760ccd418f175d6d9a2b30d00a1b566c3d5cf6de9087d39"}
Sep 18 16:22:10.535965 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:10.535927    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:22:10.537208 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:10.537174    2546 patch_prober.go:28] interesting pod/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 16:22:10.537276 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:10.537234    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj" podUID="a2036874-8eb8-4d66-bf00-b0d140e9a197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 16:22:11.487667 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.487597    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk" podUID="4adfef27-ceca-4f3d-9f21-07f76551f803" containerName="main" containerID="cri-o://30f4f93563f7440e22b4fb58ab6379dd65072c0b826ffa78eab0ff4e656172e0" gracePeriod=45
Sep 18 16:22:11.536643 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.536609    2546 patch_prober.go:28] interesting pod/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 16:22:11.536956 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.536667    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj" podUID="a2036874-8eb8-4d66-bf00-b0d140e9a197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 16:22:11.715264 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.715232    2546 generic.go:356] "Generic (PLEG): container finished" podID="4adfef27-ceca-4f3d-9f21-07f76551f803" containerID="30f4f93563f7440e22b4fb58ab6379dd65072c0b826ffa78eab0ff4e656172e0" exitCode=0
Sep 18 16:22:11.715438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.715310    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk" event={"ID":"4adfef27-ceca-4f3d-9f21-07f76551f803","Type":"ContainerDied","Data":"30f4f93563f7440e22b4fb58ab6379dd65072c0b826ffa78eab0ff4e656172e0"}
Sep 18 16:22:11.772382 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.772359    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:22:11.874782 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.874743    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4adfef27-ceca-4f3d-9f21-07f76551f803-tls-certs\") pod \"4adfef27-ceca-4f3d-9f21-07f76551f803\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") "
Sep 18 16:22:11.874958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.874827    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-tmp-dir\") pod \"4adfef27-ceca-4f3d-9f21-07f76551f803\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") "
Sep 18 16:22:11.874958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.874847    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-model-cache\") pod \"4adfef27-ceca-4f3d-9f21-07f76551f803\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") "
Sep 18 16:22:11.874958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.874917    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-home\") pod \"4adfef27-ceca-4f3d-9f21-07f76551f803\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") "
Sep 18 16:22:11.874958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.874939    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-dshm\") pod \"4adfef27-ceca-4f3d-9f21-07f76551f803\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") "
Sep 18 16:22:11.875280 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.874971    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgfvt\" (UniqueName: \"kubernetes.io/projected/4adfef27-ceca-4f3d-9f21-07f76551f803-kube-api-access-dgfvt\") pod \"4adfef27-ceca-4f3d-9f21-07f76551f803\" (UID: \"4adfef27-ceca-4f3d-9f21-07f76551f803\") "
Sep 18 16:22:11.875280 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.875169    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4adfef27-ceca-4f3d-9f21-07f76551f803" (UID: "4adfef27-ceca-4f3d-9f21-07f76551f803"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:11.875280 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.875236    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-home" (OuterVolumeSpecName: "home") pod "4adfef27-ceca-4f3d-9f21-07f76551f803" (UID: "4adfef27-ceca-4f3d-9f21-07f76551f803"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:11.875280 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.875182    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-model-cache" (OuterVolumeSpecName: "model-cache") pod "4adfef27-ceca-4f3d-9f21-07f76551f803" (UID: "4adfef27-ceca-4f3d-9f21-07f76551f803"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:11.875492 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.875463    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:11.875492 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.875483    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:11.875602 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.875496    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:11.892054 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.881168    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-dshm" (OuterVolumeSpecName: "dshm") pod "4adfef27-ceca-4f3d-9f21-07f76551f803" (UID: "4adfef27-ceca-4f3d-9f21-07f76551f803"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:11.892054 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.886238    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4adfef27-ceca-4f3d-9f21-07f76551f803-kube-api-access-dgfvt" (OuterVolumeSpecName: "kube-api-access-dgfvt") pod "4adfef27-ceca-4f3d-9f21-07f76551f803" (UID: "4adfef27-ceca-4f3d-9f21-07f76551f803"). InnerVolumeSpecName "kube-api-access-dgfvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:22:11.892386 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.892121    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4adfef27-ceca-4f3d-9f21-07f76551f803-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4adfef27-ceca-4f3d-9f21-07f76551f803" (UID: "4adfef27-ceca-4f3d-9f21-07f76551f803"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:22:11.976158 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.976121    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4adfef27-ceca-4f3d-9f21-07f76551f803-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:11.976158 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.976155    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgfvt\" (UniqueName: \"kubernetes.io/projected/4adfef27-ceca-4f3d-9f21-07f76551f803-kube-api-access-dgfvt\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:11.976158 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:11.976167    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4adfef27-ceca-4f3d-9f21-07f76551f803-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:12.536957 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:12.536919    2546 patch_prober.go:28] interesting pod/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 16:22:12.537378 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:12.536989    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj" podUID="a2036874-8eb8-4d66-bf00-b0d140e9a197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 16:22:12.723830 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:12.723793    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"
Sep 18 16:22:12.724012 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:12.723835    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk" event={"ID":"4adfef27-ceca-4f3d-9f21-07f76551f803","Type":"ContainerDied","Data":"f17aa4c74f8b1d7c173bd163adeeeb7330868e3ea6eaef4632ee0bf5c4579e8a"}
Sep 18 16:22:12.724012 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:12.723897    2546 scope.go:117] "RemoveContainer" containerID="30f4f93563f7440e22b4fb58ab6379dd65072c0b826ffa78eab0ff4e656172e0"
Sep 18 16:22:12.771567 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:12.771530    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"]
Sep 18 16:22:12.783181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:12.783147    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-eef18d/autores-model-eef18d-kserve-69b45d847-l76nk"]
Sep 18 16:22:13.536200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:13.536168    2546 patch_prober.go:28] interesting pod/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 18 16:22:13.536385 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:13.536221    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj" podUID="a2036874-8eb8-4d66-bf00-b0d140e9a197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 18 16:22:13.674211 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:13.674176    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7"]
Sep 18 16:22:13.693048 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:13.689780    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11860c1b-tc9q7"]
Sep 18 16:22:13.846469 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:13.846382    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"]
Sep 18 16:22:13.846615 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:13.846565    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj" podUID="a2036874-8eb8-4d66-bf00-b0d140e9a197" containerName="istio-proxy" containerID="cri-o://d0f1fdbddd6cee452760ccd418f175d6d9a2b30d00a1b566c3d5cf6de9087d39" gracePeriod=30
Sep 18 16:22:14.608040 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:14.607983    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b9f6c0c-952f-49e1-81a0-ff0a791134ec" path="/var/lib/kubelet/pods/0b9f6c0c-952f-49e1-81a0-ff0a791134ec/volumes"
Sep 18 16:22:14.608458 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:14.608441    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4adfef27-ceca-4f3d-9f21-07f76551f803" path="/var/lib/kubelet/pods/4adfef27-ceca-4f3d-9f21-07f76551f803/volumes"
Sep 18 16:22:20.086202 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.086167    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:22:20.396514 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.396489    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:22:20.473314 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.473283    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-credential-socket\") pod \"a2036874-8eb8-4d66-bf00-b0d140e9a197\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") "
Sep 18 16:22:20.473488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.473350    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-token\") pod \"a2036874-8eb8-4d66-bf00-b0d140e9a197\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") "
Sep 18 16:22:20.473488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.473376    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2036874-8eb8-4d66-bf00-b0d140e9a197-istiod-ca-cert\") pod \"a2036874-8eb8-4d66-bf00-b0d140e9a197\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") "
Sep 18 16:22:20.473488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.473428    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-podinfo\") pod \"a2036874-8eb8-4d66-bf00-b0d140e9a197\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") "
Sep 18 16:22:20.473488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.473456    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-workload-certs\") pod \"a2036874-8eb8-4d66-bf00-b0d140e9a197\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") "
Sep 18 16:22:20.473488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.473483    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-data\") pod \"a2036874-8eb8-4d66-bf00-b0d140e9a197\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") "
Sep 18 16:22:20.473733 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.473513    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g5zr\" (UniqueName: \"kubernetes.io/projected/a2036874-8eb8-4d66-bf00-b0d140e9a197-kube-api-access-5g5zr\") pod \"a2036874-8eb8-4d66-bf00-b0d140e9a197\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") "
Sep 18 16:22:20.473733 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.473565    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-workload-socket\") pod \"a2036874-8eb8-4d66-bf00-b0d140e9a197\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") "
Sep 18 16:22:20.473733 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.473654    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-envoy\") pod \"a2036874-8eb8-4d66-bf00-b0d140e9a197\" (UID: \"a2036874-8eb8-4d66-bf00-b0d140e9a197\") "
Sep 18 16:22:20.473733 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.473716    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-data" (OuterVolumeSpecName: "istio-data") pod "a2036874-8eb8-4d66-bf00-b0d140e9a197" (UID: "a2036874-8eb8-4d66-bf00-b0d140e9a197"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:20.473895 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.473734    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2036874-8eb8-4d66-bf00-b0d140e9a197-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a2036874-8eb8-4d66-bf00-b0d140e9a197" (UID: "a2036874-8eb8-4d66-bf00-b0d140e9a197"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:22:20.473895 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.473760    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a2036874-8eb8-4d66-bf00-b0d140e9a197" (UID: "a2036874-8eb8-4d66-bf00-b0d140e9a197"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:20.473990 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.473923    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a2036874-8eb8-4d66-bf00-b0d140e9a197" (UID: "a2036874-8eb8-4d66-bf00-b0d140e9a197"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:20.474146 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.474096    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a2036874-8eb8-4d66-bf00-b0d140e9a197" (UID: "a2036874-8eb8-4d66-bf00-b0d140e9a197"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:20.474288 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.474182    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2036874-8eb8-4d66-bf00-b0d140e9a197-istiod-ca-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:20.474288 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.474201    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-workload-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:20.474288 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.474213    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-data\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:20.474288 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.474225    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-workload-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:20.476102 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.476075    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a2036874-8eb8-4d66-bf00-b0d140e9a197" (UID: "a2036874-8eb8-4d66-bf00-b0d140e9a197"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:22:20.476188 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.476146    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-token" (OuterVolumeSpecName: "istio-token") pod "a2036874-8eb8-4d66-bf00-b0d140e9a197" (UID: "a2036874-8eb8-4d66-bf00-b0d140e9a197"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:22:20.476238 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.476201    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2036874-8eb8-4d66-bf00-b0d140e9a197-kube-api-access-5g5zr" (OuterVolumeSpecName: "kube-api-access-5g5zr") pod "a2036874-8eb8-4d66-bf00-b0d140e9a197" (UID: "a2036874-8eb8-4d66-bf00-b0d140e9a197"). InnerVolumeSpecName "kube-api-access-5g5zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:22:20.476366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.476342    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a2036874-8eb8-4d66-bf00-b0d140e9a197" (UID: "a2036874-8eb8-4d66-bf00-b0d140e9a197"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:20.575283 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.575252    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-token\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:20.575283 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.575278    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-podinfo\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:20.575283 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.575287    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g5zr\" (UniqueName: \"kubernetes.io/projected/a2036874-8eb8-4d66-bf00-b0d140e9a197-kube-api-access-5g5zr\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:20.575499 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.575298    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-istio-envoy\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:20.575499 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.575306    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2036874-8eb8-4d66-bf00-b0d140e9a197-credential-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:20.788405 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.788374    2546 generic.go:356] "Generic (PLEG): container finished" podID="a2036874-8eb8-4d66-bf00-b0d140e9a197" containerID="d0f1fdbddd6cee452760ccd418f175d6d9a2b30d00a1b566c3d5cf6de9087d39" exitCode=0
Sep 18 16:22:20.788593 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.788432    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"
Sep 18 16:22:20.788593 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.788453    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj" event={"ID":"a2036874-8eb8-4d66-bf00-b0d140e9a197","Type":"ContainerDied","Data":"d0f1fdbddd6cee452760ccd418f175d6d9a2b30d00a1b566c3d5cf6de9087d39"}
Sep 18 16:22:20.788593 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.788528    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj" event={"ID":"a2036874-8eb8-4d66-bf00-b0d140e9a197","Type":"ContainerDied","Data":"ed75b5de95c8e0b947b9ac7fc6edc6cf9f6709603ec6b079062df42de3898858"}
Sep 18 16:22:20.788593 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.788556    2546 scope.go:117] "RemoveContainer" containerID="d0f1fdbddd6cee452760ccd418f175d6d9a2b30d00a1b566c3d5cf6de9087d39"
Sep 18 16:22:20.796998 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.796980    2546 scope.go:117] "RemoveContainer" containerID="4e72a64fbfe72a9d8584c68849712a2017613d6c2765fec369680a4955d9e71c"
Sep 18 16:22:20.804533 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.804516    2546 scope.go:117] "RemoveContainer" containerID="d0f1fdbddd6cee452760ccd418f175d6d9a2b30d00a1b566c3d5cf6de9087d39"
Sep 18 16:22:20.804800 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:22:20.804779    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f1fdbddd6cee452760ccd418f175d6d9a2b30d00a1b566c3d5cf6de9087d39\": container with ID starting with d0f1fdbddd6cee452760ccd418f175d6d9a2b30d00a1b566c3d5cf6de9087d39 not found: ID does not exist" containerID="d0f1fdbddd6cee452760ccd418f175d6d9a2b30d00a1b566c3d5cf6de9087d39"
Sep 18 16:22:20.804871 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.804812    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f1fdbddd6cee452760ccd418f175d6d9a2b30d00a1b566c3d5cf6de9087d39"} err="failed to get container status \"d0f1fdbddd6cee452760ccd418f175d6d9a2b30d00a1b566c3d5cf6de9087d39\": rpc error: code = NotFound desc = could not find container \"d0f1fdbddd6cee452760ccd418f175d6d9a2b30d00a1b566c3d5cf6de9087d39\": container with ID starting with d0f1fdbddd6cee452760ccd418f175d6d9a2b30d00a1b566c3d5cf6de9087d39 not found: ID does not exist"
Sep 18 16:22:20.804871 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.804833    2546 scope.go:117] "RemoveContainer" containerID="4e72a64fbfe72a9d8584c68849712a2017613d6c2765fec369680a4955d9e71c"
Sep 18 16:22:20.805127 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:22:20.805110    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e72a64fbfe72a9d8584c68849712a2017613d6c2765fec369680a4955d9e71c\": container with ID starting with 4e72a64fbfe72a9d8584c68849712a2017613d6c2765fec369680a4955d9e71c not found: ID does not exist" containerID="4e72a64fbfe72a9d8584c68849712a2017613d6c2765fec369680a4955d9e71c"
Sep 18 16:22:20.805172 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.805132    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e72a64fbfe72a9d8584c68849712a2017613d6c2765fec369680a4955d9e71c"} err="failed to get container status \"4e72a64fbfe72a9d8584c68849712a2017613d6c2765fec369680a4955d9e71c\": rpc error: code = NotFound desc = could not find container \"4e72a64fbfe72a9d8584c68849712a2017613d6c2765fec369680a4955d9e71c\": container with ID starting with 4e72a64fbfe72a9d8584c68849712a2017613d6c2765fec369680a4955d9e71c not found: ID does not exist"
Sep 18 16:22:20.813756 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.813736    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"]
Sep 18 16:22:20.818285 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:20.818266    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-11860c1b-openshift-default-85fb9f5f44-7ljwj"]
Sep 18 16:22:22.603983 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:22.603948    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2036874-8eb8-4d66-bf00-b0d140e9a197" path="/var/lib/kubelet/pods/a2036874-8eb8-4d66-bf00-b0d140e9a197/volumes"
Sep 18 16:22:27.822646 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:27.822604    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"]
Sep 18 16:22:27.823108 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:27.822890    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy" containerID="cri-o://6ded38bfd83180008ea092468d2b9bea84345ad09fe0ea7cba49f7308c849486" gracePeriod=30
Sep 18 16:22:33.078864 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.078825    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:22:33.211689 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.211658    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-podinfo\") pod \"4f6ec037-867b-4f80-a175-595eb9355fcd\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") "
Sep 18 16:22:33.211689 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.211694    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmfp8\" (UniqueName: \"kubernetes.io/projected/4f6ec037-867b-4f80-a175-595eb9355fcd-kube-api-access-gmfp8\") pod \"4f6ec037-867b-4f80-a175-595eb9355fcd\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") "
Sep 18 16:22:33.211957 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.211755    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-workload-socket\") pod \"4f6ec037-867b-4f80-a175-595eb9355fcd\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") "
Sep 18 16:22:33.211957 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.211785    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-credential-socket\") pod \"4f6ec037-867b-4f80-a175-595eb9355fcd\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") "
Sep 18 16:22:33.211957 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.211807    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-token\") pod \"4f6ec037-867b-4f80-a175-595eb9355fcd\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") "
Sep 18 16:22:33.212122 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.211975    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6ec037-867b-4f80-a175-595eb9355fcd-istiod-ca-cert\") pod \"4f6ec037-867b-4f80-a175-595eb9355fcd\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") "
Sep 18 16:22:33.212122 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.212007    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-envoy\") pod \"4f6ec037-867b-4f80-a175-595eb9355fcd\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") "
Sep 18 16:22:33.212122 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.212057    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-workload-certs\") pod \"4f6ec037-867b-4f80-a175-595eb9355fcd\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") "
Sep 18 16:22:33.212278 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.212140    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-data\") pod \"4f6ec037-867b-4f80-a175-595eb9355fcd\" (UID: \"4f6ec037-867b-4f80-a175-595eb9355fcd\") "
Sep 18 16:22:33.212278 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.212144    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4f6ec037-867b-4f80-a175-595eb9355fcd" (UID: "4f6ec037-867b-4f80-a175-595eb9355fcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:33.212278 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.212164    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4f6ec037-867b-4f80-a175-595eb9355fcd" (UID: "4f6ec037-867b-4f80-a175-595eb9355fcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:33.212432 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.212328    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4f6ec037-867b-4f80-a175-595eb9355fcd" (UID: "4f6ec037-867b-4f80-a175-595eb9355fcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:33.212621 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.212554    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6ec037-867b-4f80-a175-595eb9355fcd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4f6ec037-867b-4f80-a175-595eb9355fcd" (UID: "4f6ec037-867b-4f80-a175-595eb9355fcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:22:33.212621 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.212564    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-data" (OuterVolumeSpecName: "istio-data") pod "4f6ec037-867b-4f80-a175-595eb9355fcd" (UID: "4f6ec037-867b-4f80-a175-595eb9355fcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:33.212621 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.212565    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-workload-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:33.212621 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.212598    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-workload-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:33.212621 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.212617    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-credential-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:33.214226 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.214196    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4f6ec037-867b-4f80-a175-595eb9355fcd" (UID: "4f6ec037-867b-4f80-a175-595eb9355fcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:22:33.214417 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.214392    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f6ec037-867b-4f80-a175-595eb9355fcd" (UID: "4f6ec037-867b-4f80-a175-595eb9355fcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:22:33.214511 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.214491    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6ec037-867b-4f80-a175-595eb9355fcd-kube-api-access-gmfp8" (OuterVolumeSpecName: "kube-api-access-gmfp8") pod "4f6ec037-867b-4f80-a175-595eb9355fcd" (UID: "4f6ec037-867b-4f80-a175-595eb9355fcd"). InnerVolumeSpecName "kube-api-access-gmfp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:22:33.214567 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.214515    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4f6ec037-867b-4f80-a175-595eb9355fcd" (UID: "4f6ec037-867b-4f80-a175-595eb9355fcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:22:33.313724 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.313693    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f6ec037-867b-4f80-a175-595eb9355fcd-istiod-ca-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:33.313724 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.313721    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-envoy\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:33.313724 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.313732    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-data\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:33.313968 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.313742    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-podinfo\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:33.313968 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.313754    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmfp8\" (UniqueName: \"kubernetes.io/projected/4f6ec037-867b-4f80-a175-595eb9355fcd-kube-api-access-gmfp8\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:33.313968 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.313766    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f6ec037-867b-4f80-a175-595eb9355fcd-istio-token\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:22:33.868148 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.868116    2546 generic.go:356] "Generic (PLEG): container finished" podID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerID="6ded38bfd83180008ea092468d2b9bea84345ad09fe0ea7cba49f7308c849486" exitCode=0
Sep 18 16:22:33.868352 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.868178    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"
Sep 18 16:22:33.868352 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.868199    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" event={"ID":"4f6ec037-867b-4f80-a175-595eb9355fcd","Type":"ContainerDied","Data":"6ded38bfd83180008ea092468d2b9bea84345ad09fe0ea7cba49f7308c849486"}
Sep 18 16:22:33.868352 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.868265    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f" event={"ID":"4f6ec037-867b-4f80-a175-595eb9355fcd","Type":"ContainerDied","Data":"8019e8813e8f5e665fd5bdd34822f3bdd6cdc5bb9d1ede15841f3cc03c01c9c3"}
Sep 18 16:22:33.868352 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.868293    2546 scope.go:117] "RemoveContainer" containerID="6ded38bfd83180008ea092468d2b9bea84345ad09fe0ea7cba49f7308c849486"
Sep 18 16:22:33.877813 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.877791    2546 scope.go:117] "RemoveContainer" containerID="aa4b7b8f285c65643518b2fc64d0c66508528a6290aed1dfda72323163f1dddb"
Sep 18 16:22:33.888017 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.888001    2546 scope.go:117] "RemoveContainer" containerID="6ded38bfd83180008ea092468d2b9bea84345ad09fe0ea7cba49f7308c849486"
Sep 18 16:22:33.888314 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:22:33.888296    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ded38bfd83180008ea092468d2b9bea84345ad09fe0ea7cba49f7308c849486\": container with ID starting with 6ded38bfd83180008ea092468d2b9bea84345ad09fe0ea7cba49f7308c849486 not found: ID does not exist" containerID="6ded38bfd83180008ea092468d2b9bea84345ad09fe0ea7cba49f7308c849486"
Sep 18 16:22:33.888416 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.888324    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ded38bfd83180008ea092468d2b9bea84345ad09fe0ea7cba49f7308c849486"} err="failed to get container status \"6ded38bfd83180008ea092468d2b9bea84345ad09fe0ea7cba49f7308c849486\": rpc error: code = NotFound desc = could not find container \"6ded38bfd83180008ea092468d2b9bea84345ad09fe0ea7cba49f7308c849486\": container with ID starting with 6ded38bfd83180008ea092468d2b9bea84345ad09fe0ea7cba49f7308c849486 not found: ID does not exist"
Sep 18 16:22:33.888416 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.888347    2546 scope.go:117] "RemoveContainer" containerID="aa4b7b8f285c65643518b2fc64d0c66508528a6290aed1dfda72323163f1dddb"
Sep 18 16:22:33.888630 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:22:33.888607    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa4b7b8f285c65643518b2fc64d0c66508528a6290aed1dfda72323163f1dddb\": container with ID starting with aa4b7b8f285c65643518b2fc64d0c66508528a6290aed1dfda72323163f1dddb not found: ID does not exist" containerID="aa4b7b8f285c65643518b2fc64d0c66508528a6290aed1dfda72323163f1dddb"
Sep 18 16:22:33.888668 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.888636    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa4b7b8f285c65643518b2fc64d0c66508528a6290aed1dfda72323163f1dddb"} err="failed to get container status \"aa4b7b8f285c65643518b2fc64d0c66508528a6290aed1dfda72323163f1dddb\": rpc error: code = NotFound desc = could not find container \"aa4b7b8f285c65643518b2fc64d0c66508528a6290aed1dfda72323163f1dddb\": container with ID starting with aa4b7b8f285c65643518b2fc64d0c66508528a6290aed1dfda72323163f1dddb not found: ID does not exist"
Sep 18 16:22:33.907640 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.907616    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"]
Sep 18 16:22:33.919500 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:33.919478    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-eef18d-openshift-default-5f67b8d696-9955f"]
Sep 18 16:22:34.603015 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:34.602976    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" path="/var/lib/kubelet/pods/4f6ec037-867b-4f80-a175-595eb9355fcd/volumes"
Sep 18 16:22:48.466087 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:22:48.466053    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:23:06.761968 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.761932    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"]
Sep 18 16:23:06.762354 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762340    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4adfef27-ceca-4f3d-9f21-07f76551f803" containerName="main"
Sep 18 16:23:06.762400 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762356    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4adfef27-ceca-4f3d-9f21-07f76551f803" containerName="main"
Sep 18 16:23:06.762400 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762363    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2036874-8eb8-4d66-bf00-b0d140e9a197" containerName="istio-proxy"
Sep 18 16:23:06.762400 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762368    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2036874-8eb8-4d66-bf00-b0d140e9a197" containerName="istio-proxy"
Sep 18 16:23:06.762400 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762384    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy"
Sep 18 16:23:06.762400 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762389    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy"
Sep 18 16:23:06.762400 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762395    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b9f6c0c-952f-49e1-81a0-ff0a791134ec" containerName="revoke-keys"
Sep 18 16:23:06.762400 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762400    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b9f6c0c-952f-49e1-81a0-ff0a791134ec" containerName="revoke-keys"
Sep 18 16:23:06.762630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762407    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2036874-8eb8-4d66-bf00-b0d140e9a197" containerName="istio-proxy"
Sep 18 16:23:06.762630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762412    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2036874-8eb8-4d66-bf00-b0d140e9a197" containerName="istio-proxy"
Sep 18 16:23:06.762630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762419    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy"
Sep 18 16:23:06.762630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762424    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy"
Sep 18 16:23:06.762630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762488    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4adfef27-ceca-4f3d-9f21-07f76551f803" containerName="main"
Sep 18 16:23:06.762630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762496    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy"
Sep 18 16:23:06.762630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762502    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f6ec037-867b-4f80-a175-595eb9355fcd" containerName="istio-proxy"
Sep 18 16:23:06.762630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762508    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2036874-8eb8-4d66-bf00-b0d140e9a197" containerName="istio-proxy"
Sep 18 16:23:06.762630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762515    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2036874-8eb8-4d66-bf00-b0d140e9a197" containerName="istio-proxy"
Sep 18 16:23:06.762630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.762522    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b9f6c0c-952f-49e1-81a0-ff0a791134ec" containerName="revoke-keys"
Sep 18 16:23:06.766783 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.766768    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:06.770304 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.770284    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4f2d8a\"/\"premium-w4-4f2d8a-kserve-self-signed-certs\""
Sep 18 16:23:06.804038 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.803995    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"]
Sep 18 16:23:06.939875 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.939824    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-model-cache\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:06.940108 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.939899    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-home\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:06.940108 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.939971    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cca791-527d-4558-86b8-a0833983da93-tls-certs\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:06.940108 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.940007    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gmfh\" (UniqueName: \"kubernetes.io/projected/a8cca791-527d-4558-86b8-a0833983da93-kube-api-access-6gmfh\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:06.940108 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.940073    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-tmp-dir\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:06.940356 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:06.940129    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-dshm\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.040901 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.040839    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-model-cache\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.040901 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.040880    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-home\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.041121 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.041017    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cca791-527d-4558-86b8-a0833983da93-tls-certs\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.041121 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.041111    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gmfh\" (UniqueName: \"kubernetes.io/projected/a8cca791-527d-4558-86b8-a0833983da93-kube-api-access-6gmfh\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.041233 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.041157    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-tmp-dir\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.041233 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.041213    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-dshm\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.041326 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.041270    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-home\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.041326 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.041302    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-model-cache\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.041480 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.041454    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-tmp-dir\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.043247 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.043224    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-dshm\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.043404 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.043389    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cca791-527d-4558-86b8-a0833983da93-tls-certs\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.052676 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.052655    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gmfh\" (UniqueName: \"kubernetes.io/projected/a8cca791-527d-4558-86b8-a0833983da93-kube-api-access-6gmfh\") pod \"premium-w4-4f2d8a-kserve-6558b99857-sj6ft\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") " pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.075845 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.075823    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:07.363575 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:07.363545    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"]
Sep 18 16:23:07.364391 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:23:07.364363    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8cca791_527d_4558_86b8_a0833983da93.slice/crio-c9700932e2559cf9aed9ec3274f6ed12748b04832479e2ac18c8248f6c2ba21a WatchSource:0}: Error finding container c9700932e2559cf9aed9ec3274f6ed12748b04832479e2ac18c8248f6c2ba21a: Status 404 returned error can't find the container with id c9700932e2559cf9aed9ec3274f6ed12748b04832479e2ac18c8248f6c2ba21a
Sep 18 16:23:08.064169 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:08.064131    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft" event={"ID":"a8cca791-527d-4558-86b8-a0833983da93","Type":"ContainerStarted","Data":"7dddcaafbdd2195e539f9ba3cfa51606281dd06d71a98e4933ae314fcc9f06ff"}
Sep 18 16:23:08.064517 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:08.064180    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft" event={"ID":"a8cca791-527d-4558-86b8-a0833983da93","Type":"ContainerStarted","Data":"c9700932e2559cf9aed9ec3274f6ed12748b04832479e2ac18c8248f6c2ba21a"}
Sep 18 16:23:08.096952 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:08.096906    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft" podStartSLOduration=2.096891083 podStartE2EDuration="2.096891083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:23:08.095140849 +0000 UTC m=+1084.003921198" watchObservedRunningTime="2026-09-18 16:23:08.096891083 +0000 UTC m=+1084.005671433"
Sep 18 16:23:12.361254 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:12.361220    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:23:17.076525 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:17.076492    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:17.077147 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:17.076543    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:17.089373 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:17.089344    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:17.123358 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:17.123334    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:23:25.892854 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:25.892824    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"]
Sep 18 16:23:25.896741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:25.896726    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:25.900238 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:25.900219    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1eec1d\"/\"premium-w0-1eec1d-kserve-self-signed-certs\""
Sep 18 16:23:25.919482 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:25.919460    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-model-cache\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:25.919576 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:25.919485    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw5cx\" (UniqueName: \"kubernetes.io/projected/d7318181-8bda-493f-bc63-0bf78a370280-kube-api-access-qw5cx\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:25.919576 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:25.919509    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-dshm\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:25.919576 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:25.919526    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7318181-8bda-493f-bc63-0bf78a370280-tls-certs\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:25.919734 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:25.919652    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-home\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:25.919734 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:25.919719    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-tmp-dir\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:25.924978 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:25.924959    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"]
Sep 18 16:23:26.021010 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.020974    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-model-cache\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:26.021306 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.021286    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw5cx\" (UniqueName: \"kubernetes.io/projected/d7318181-8bda-493f-bc63-0bf78a370280-kube-api-access-qw5cx\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:26.021501 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.021473    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-dshm\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:26.021645 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.021627    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7318181-8bda-493f-bc63-0bf78a370280-tls-certs\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:26.021935 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.021918    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-home\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:26.023267 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.023243    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-tmp-dir\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:26.023859 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.023835    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-tmp-dir\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:26.024085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.022476    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-home\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:26.024793 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.024771    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-model-cache\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:26.027435 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.027413    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-dshm\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:26.029817 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.029793    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7318181-8bda-493f-bc63-0bf78a370280-tls-certs\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:26.035047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.034998    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw5cx\" (UniqueName: \"kubernetes.io/projected/d7318181-8bda-493f-bc63-0bf78a370280-kube-api-access-qw5cx\") pod \"premium-w0-1eec1d-kserve-685584b597-tmq5n\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") " pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:26.205976 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.205950    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:26.488618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:26.488449    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"]
Sep 18 16:23:26.490756 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:23:26.490721    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7318181_8bda_493f_bc63_0bf78a370280.slice/crio-e6aed5a2fe04ec27053648a8fb9cd331b52bc0501e7429f9f7df90be65123bf7 WatchSource:0}: Error finding container e6aed5a2fe04ec27053648a8fb9cd331b52bc0501e7429f9f7df90be65123bf7: Status 404 returned error can't find the container with id e6aed5a2fe04ec27053648a8fb9cd331b52bc0501e7429f9f7df90be65123bf7
Sep 18 16:23:27.169300 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:27.169266    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n" event={"ID":"d7318181-8bda-493f-bc63-0bf78a370280","Type":"ContainerStarted","Data":"16df1713faa603163b9b08e792d2b1532a50de1ec92fdf74eae3ed9c0fa67821"}
Sep 18 16:23:27.169694 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:27.169311    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n" event={"ID":"d7318181-8bda-493f-bc63-0bf78a370280","Type":"ContainerStarted","Data":"e6aed5a2fe04ec27053648a8fb9cd331b52bc0501e7429f9f7df90be65123bf7"}
Sep 18 16:23:27.194206 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:27.194158    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n" podStartSLOduration=2.194143326 podStartE2EDuration="2.194143326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:23:27.191748974 +0000 UTC m=+1103.100529327" watchObservedRunningTime="2026-09-18 16:23:27.194143326 +0000 UTC m=+1103.102923676"
Sep 18 16:23:33.671196 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:33.671159    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:23:36.206985 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:36.206948    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:36.207393 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:36.206999    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:36.218952 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:36.218925    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:36.234940 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:36.234917    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:23:57.651609 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:23:57.651572    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:24:22.404866 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:22.404830    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:24:26.264516 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.264482    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5"]
Sep 18 16:24:26.269163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.269127    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5"
Sep 18 16:24:26.273320 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.273293    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:24:26.273740 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.273720    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:24:26.274002 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.273979    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"]
Sep 18 16:24:26.278191 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.278171    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.282479 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.282459    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f22789\"/\"kube-root-ca.crt\""
Sep 18 16:24:26.283462 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.283438    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f22789\"/\"test-model-f22789-kserve-self-signed-certs\""
Sep 18 16:24:26.283868 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.283805    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f22789\"/\"openshift-service-ca.crt\""
Sep 18 16:24:26.285408 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.285391    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f22789\"/\"default-dockercfg-94xcv\""
Sep 18 16:24:26.288833 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.288813    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5"]
Sep 18 16:24:26.292262 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.292240    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"]
Sep 18 16:24:26.397121 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.397089    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-dshm\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.397283 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.397127    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-home\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.397283 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.397192    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24rbk\" (UniqueName: \"kubernetes.io/projected/6db7a1cf-4aa6-4c88-9778-9da8485536a2-kube-api-access-24rbk\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.397403 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.397300    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca25600-8a4a-460a-981d-2bab76a958f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5\" (UID: \"8ca25600-8a4a-460a-981d-2bab76a958f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5"
Sep 18 16:24:26.397490 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.397472    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db7a1cf-4aa6-4c88-9778-9da8485536a2-tls-certs\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.397602 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.397514    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-model-cache\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.397602 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.397539    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-tmp-dir\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.498206 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.498173    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db7a1cf-4aa6-4c88-9778-9da8485536a2-tls-certs\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.498365 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.498211    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-model-cache\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.498365 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.498232    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-tmp-dir\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.498365 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.498339    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-dshm\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.498507 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.498370    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-home\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.498507 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.498411    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24rbk\" (UniqueName: \"kubernetes.io/projected/6db7a1cf-4aa6-4c88-9778-9da8485536a2-kube-api-access-24rbk\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.498507 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.498472    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca25600-8a4a-460a-981d-2bab76a958f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5\" (UID: \"8ca25600-8a4a-460a-981d-2bab76a958f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5"
Sep 18 16:24:26.498668 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.498594    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-tmp-dir\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.498827 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.498801    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-model-cache\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.498827 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.498817    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-home\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.499200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.499180    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca25600-8a4a-460a-981d-2bab76a958f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5\" (UID: \"8ca25600-8a4a-460a-981d-2bab76a958f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5"
Sep 18 16:24:26.500415 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.500398    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-dshm\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.500810 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.500782    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db7a1cf-4aa6-4c88-9778-9da8485536a2-tls-certs\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.507949 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.507928    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24rbk\" (UniqueName: \"kubernetes.io/projected/6db7a1cf-4aa6-4c88-9778-9da8485536a2-kube-api-access-24rbk\") pod \"test-model-f22789-kserve-7b46b4d8b9-hgbbs\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") " pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.585716 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.585625    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5"
Sep 18 16:24:26.593406 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.593379    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:26.869754 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.869725    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5"]
Sep 18 16:24:26.871922 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:24:26.871894    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca25600_8a4a_460a_981d_2bab76a958f9.slice/crio-f0e96fd1d73d687b357568d0497cc09707b4b3d64a6d1501c5c785b22e7c4c57 WatchSource:0}: Error finding container f0e96fd1d73d687b357568d0497cc09707b4b3d64a6d1501c5c785b22e7c4c57: Status 404 returned error can't find the container with id f0e96fd1d73d687b357568d0497cc09707b4b3d64a6d1501c5c785b22e7c4c57
Sep 18 16:24:26.872359 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:26.872341    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"]
Sep 18 16:24:26.872889 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:24:26.872865    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6db7a1cf_4aa6_4c88_9778_9da8485536a2.slice/crio-177c8f6b804a8265ea3d1eecd1af0b7b40074812ea66a187c47508160dcc7f5c WatchSource:0}: Error finding container 177c8f6b804a8265ea3d1eecd1af0b7b40074812ea66a187c47508160dcc7f5c: Status 404 returned error can't find the container with id 177c8f6b804a8265ea3d1eecd1af0b7b40074812ea66a187c47508160dcc7f5c
Sep 18 16:24:27.517746 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:27.517669    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5" event={"ID":"8ca25600-8a4a-460a-981d-2bab76a958f9","Type":"ContainerStarted","Data":"1c67273ed91e49b7ec71e243e6bc3f765334fe69eb68339eb19be0db0ae5a65e"}
Sep 18 16:24:27.517746 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:27.517722    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5" event={"ID":"8ca25600-8a4a-460a-981d-2bab76a958f9","Type":"ContainerStarted","Data":"f0e96fd1d73d687b357568d0497cc09707b4b3d64a6d1501c5c785b22e7c4c57"}
Sep 18 16:24:27.519274 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:27.519240    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs" event={"ID":"6db7a1cf-4aa6-4c88-9778-9da8485536a2","Type":"ContainerStarted","Data":"b864b2b111b479710c7f9bdfd4007dee38d4aa4d778a140c5034af707e5e8966"}
Sep 18 16:24:27.519380 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:27.519287    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs" event={"ID":"6db7a1cf-4aa6-4c88-9778-9da8485536a2","Type":"ContainerStarted","Data":"177c8f6b804a8265ea3d1eecd1af0b7b40074812ea66a187c47508160dcc7f5c"}
Sep 18 16:24:27.537098 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:27.537054    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5" podStartSLOduration=2.5370165350000002 podStartE2EDuration="2.537016535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:24:27.533764272 +0000 UTC m=+1163.442544641" watchObservedRunningTime="2026-09-18 16:24:27.537016535 +0000 UTC m=+1163.445796892"
Sep 18 16:24:27.558994 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:27.558955    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs" podStartSLOduration=1.558942117 podStartE2EDuration="1.558942117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:24:27.556094948 +0000 UTC m=+1163.464875319" watchObservedRunningTime="2026-09-18 16:24:27.558942117 +0000 UTC m=+1163.467722468"
Sep 18 16:24:28.526882 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:28.526850    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ca25600-8a4a-460a-981d-2bab76a958f9" containerID="1c67273ed91e49b7ec71e243e6bc3f765334fe69eb68339eb19be0db0ae5a65e" exitCode=0
Sep 18 16:24:28.527451 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:28.526932    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5" event={"ID":"8ca25600-8a4a-460a-981d-2bab76a958f9","Type":"ContainerDied","Data":"1c67273ed91e49b7ec71e243e6bc3f765334fe69eb68339eb19be0db0ae5a65e"}
Sep 18 16:24:29.664637 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:29.664613    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5"
Sep 18 16:24:29.833907 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:29.833832    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca25600-8a4a-460a-981d-2bab76a958f9-maas-api-service-ca\") pod \"8ca25600-8a4a-460a-981d-2bab76a958f9\" (UID: \"8ca25600-8a4a-460a-981d-2bab76a958f9\") "
Sep 18 16:24:29.834182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:29.834163    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca25600-8a4a-460a-981d-2bab76a958f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ca25600-8a4a-460a-981d-2bab76a958f9" (UID: "8ca25600-8a4a-460a-981d-2bab76a958f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:24:29.934881 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:29.934849    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ca25600-8a4a-460a-981d-2bab76a958f9-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:24:30.546849 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:30.546813    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5"
Sep 18 16:24:30.547076 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:30.546837    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5" event={"ID":"8ca25600-8a4a-460a-981d-2bab76a958f9","Type":"ContainerDied","Data":"f0e96fd1d73d687b357568d0497cc09707b4b3d64a6d1501c5c785b22e7c4c57"}
Sep 18 16:24:30.547076 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:30.546977    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0e96fd1d73d687b357568d0497cc09707b4b3d64a6d1501c5c785b22e7c4c57"
Sep 18 16:24:36.594972 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:36.594935    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:36.595360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:36.594982    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:36.607917 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:36.607891    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:37.612285 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:37.612243    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:24:41.713629 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:41.713591    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5"]
Sep 18 16:24:41.717156 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:41.717129    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-b73e26-kjwz5"]
Sep 18 16:24:42.603829 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:42.603794    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ca25600-8a4a-460a-981d-2bab76a958f9" path="/var/lib/kubelet/pods/8ca25600-8a4a-460a-981d-2bab76a958f9/volumes"
Sep 18 16:24:45.297787 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:45.297745    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld"]
Sep 18 16:24:45.298195 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:45.298172    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ca25600-8a4a-460a-981d-2bab76a958f9" containerName="revoke-keys"
Sep 18 16:24:45.298195 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:45.298185    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ca25600-8a4a-460a-981d-2bab76a958f9" containerName="revoke-keys"
Sep 18 16:24:45.298274 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:45.298249    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ca25600-8a4a-460a-981d-2bab76a958f9" containerName="revoke-keys"
Sep 18 16:24:45.301411 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:45.301393    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld"
Sep 18 16:24:45.308378 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:45.308359    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:24:45.308378 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:45.308359    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:24:45.311389 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:45.311366    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld"]
Sep 18 16:24:45.404629 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:45.404596    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcecde84-0278-45e6-ae83-d8831705d2bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-eab152b6-bdgld\" (UID: \"fcecde84-0278-45e6-ae83-d8831705d2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld"
Sep 18 16:24:45.505665 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:45.505627    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcecde84-0278-45e6-ae83-d8831705d2bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-eab152b6-bdgld\" (UID: \"fcecde84-0278-45e6-ae83-d8831705d2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld"
Sep 18 16:24:45.506314 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:45.506290    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcecde84-0278-45e6-ae83-d8831705d2bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-eab152b6-bdgld\" (UID: \"fcecde84-0278-45e6-ae83-d8831705d2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld"
Sep 18 16:24:45.611794 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:45.611705    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld"
Sep 18 16:24:45.893516 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:45.893490    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld"]
Sep 18 16:24:45.895009 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:24:45.894978    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcecde84_0278_45e6_ae83_d8831705d2bf.slice/crio-791ddb8bce594f2948a2b88fe868062e1badf6cde1c078718aec212e1a3ad501 WatchSource:0}: Error finding container 791ddb8bce594f2948a2b88fe868062e1badf6cde1c078718aec212e1a3ad501: Status 404 returned error can't find the container with id 791ddb8bce594f2948a2b88fe868062e1badf6cde1c078718aec212e1a3ad501
Sep 18 16:24:46.666244 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:46.666197    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld" event={"ID":"fcecde84-0278-45e6-ae83-d8831705d2bf","Type":"ContainerStarted","Data":"2c8a6401bf16374a8320453036c9e4e4c0d4a4223f8a54196e1e3fceb09fe460"}
Sep 18 16:24:46.666644 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:46.666266    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld" event={"ID":"fcecde84-0278-45e6-ae83-d8831705d2bf","Type":"ContainerStarted","Data":"791ddb8bce594f2948a2b88fe868062e1badf6cde1c078718aec212e1a3ad501"}
Sep 18 16:24:46.694248 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:46.694204    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld" podStartSLOduration=1.694188724 podStartE2EDuration="1.694188724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:24:46.691501792 +0000 UTC m=+1182.600282169" watchObservedRunningTime="2026-09-18 16:24:46.694188724 +0000 UTC m=+1182.602969074"
Sep 18 16:24:47.673013 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:47.672980    2546 generic.go:356] "Generic (PLEG): container finished" podID="fcecde84-0278-45e6-ae83-d8831705d2bf" containerID="2c8a6401bf16374a8320453036c9e4e4c0d4a4223f8a54196e1e3fceb09fe460" exitCode=0
Sep 18 16:24:47.673410 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:47.673062    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld" event={"ID":"fcecde84-0278-45e6-ae83-d8831705d2bf","Type":"ContainerDied","Data":"2c8a6401bf16374a8320453036c9e4e4c0d4a4223f8a54196e1e3fceb09fe460"}
Sep 18 16:24:48.821299 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:48.821278    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld"
Sep 18 16:24:48.939098 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:48.939008    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcecde84-0278-45e6-ae83-d8831705d2bf-maas-api-service-ca\") pod \"fcecde84-0278-45e6-ae83-d8831705d2bf\" (UID: \"fcecde84-0278-45e6-ae83-d8831705d2bf\") "
Sep 18 16:24:48.939312 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:48.939292    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcecde84-0278-45e6-ae83-d8831705d2bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fcecde84-0278-45e6-ae83-d8831705d2bf" (UID: "fcecde84-0278-45e6-ae83-d8831705d2bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:24:49.040362 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:49.040331    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcecde84-0278-45e6-ae83-d8831705d2bf-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:24:49.686803 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:49.686769    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld" event={"ID":"fcecde84-0278-45e6-ae83-d8831705d2bf","Type":"ContainerDied","Data":"791ddb8bce594f2948a2b88fe868062e1badf6cde1c078718aec212e1a3ad501"}
Sep 18 16:24:49.686803 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:49.686796    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld"
Sep 18 16:24:49.687051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:49.686816    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="791ddb8bce594f2948a2b88fe868062e1badf6cde1c078718aec212e1a3ad501"
Sep 18 16:24:51.263157 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:51.263121    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:24:56.049703 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:56.049661    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld"]
Sep 18 16:24:56.053566 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:56.053542    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eab152b6-bdgld"]
Sep 18 16:24:56.603347 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:24:56.603314    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcecde84-0278-45e6-ae83-d8831705d2bf" path="/var/lib/kubelet/pods/fcecde84-0278-45e6-ae83-d8831705d2bf/volumes"
Sep 18 16:25:18.058727 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:18.058691    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:25:28.856122 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.856082    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"]
Sep 18 16:25:28.857272 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.857245    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcecde84-0278-45e6-ae83-d8831705d2bf" containerName="revoke-keys"
Sep 18 16:25:28.857442 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.857431    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcecde84-0278-45e6-ae83-d8831705d2bf" containerName="revoke-keys"
Sep 18 16:25:28.857668 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.857655    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcecde84-0278-45e6-ae83-d8831705d2bf" containerName="revoke-keys"
Sep 18 16:25:28.862312 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.862284    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:28.871372 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.871339    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"]
Sep 18 16:25:28.871764 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.871524    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-72547f\"/\"noten-model-72547f-kserve-self-signed-certs\""
Sep 18 16:25:28.871764 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.871566    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-72547f\"/\"openshift-service-ca.crt\""
Sep 18 16:25:28.871764 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.871633    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-72547f\"/\"kube-root-ca.crt\""
Sep 18 16:25:28.871976 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.871904    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-72547f\"/\"default-dockercfg-ztlwk\""
Sep 18 16:25:28.970002 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.969967    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-model-cache\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:28.970163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.970016    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-tmp-dir\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:28.970163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.970074    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvvd\" (UniqueName: \"kubernetes.io/projected/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-kube-api-access-ksvvd\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:28.970163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.970120    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-tls-certs\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:28.970163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.970143    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-home\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:28.970163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:28.970163    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-dshm\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.071703 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.071672    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-model-cache\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.071862 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.071719    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-tmp-dir\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.071862 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.071747    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvvd\" (UniqueName: \"kubernetes.io/projected/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-kube-api-access-ksvvd\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.071862 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.071779    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-tls-certs\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.071862 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.071797    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-home\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.071862 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.071815    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-dshm\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.072162 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.072137    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-model-cache\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.072162 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.072150    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-tmp-dir\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.072271 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.072222    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-home\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.073932 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.073911    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-dshm\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.074251 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.074235    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-tls-certs\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.088971 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.088937    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvvd\" (UniqueName: \"kubernetes.io/projected/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-kube-api-access-ksvvd\") pod \"noten-model-72547f-kserve-7c975f8dc-l4b6z\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") " pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.185716 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.185628    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:29.474749 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.474727    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"]
Sep 18 16:25:29.476805 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:25:29.476779    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fe3ee55_fe3f_4939_989a_a2d460cf8c16.slice/crio-798a1805894f95a30d458605de3752c6d9e7b13c917ab7979c3572fe4fd12220 WatchSource:0}: Error finding container 798a1805894f95a30d458605de3752c6d9e7b13c917ab7979c3572fe4fd12220: Status 404 returned error can't find the container with id 798a1805894f95a30d458605de3752c6d9e7b13c917ab7979c3572fe4fd12220
Sep 18 16:25:29.927640 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.927555    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z" event={"ID":"5fe3ee55-fe3f-4939-989a-a2d460cf8c16","Type":"ContainerStarted","Data":"07695c9d8ee041361f82e87697f1d91aadcda1f17abb1f1579e0aaec57cad5ca"}
Sep 18 16:25:29.927640 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.927597    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z" event={"ID":"5fe3ee55-fe3f-4939-989a-a2d460cf8c16","Type":"ContainerStarted","Data":"798a1805894f95a30d458605de3752c6d9e7b13c917ab7979c3572fe4fd12220"}
Sep 18 16:25:29.952429 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:29.952382    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z" podStartSLOduration=1.952367609 podStartE2EDuration="1.952367609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:25:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:25:29.949841936 +0000 UTC m=+1225.858622286" watchObservedRunningTime="2026-09-18 16:25:29.952367609 +0000 UTC m=+1225.861147959"
Sep 18 16:25:39.186817 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:39.186773    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:39.187347 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:39.186846    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:39.198720 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:39.198696    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:40.002746 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:40.002717    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:25:47.671747 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:47.671700    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:25:52.457493 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:25:52.457461    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"]
Sep 18 16:26:04.764304 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:04.764241    2546 scope.go:117] "RemoveContainer" containerID="16e5c82804a635e70c160088111d056038956519d5fde954b04ed586928adb37"
Sep 18 16:26:07.476302 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.476259    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z" podUID="5fe3ee55-fe3f-4939-989a-a2d460cf8c16" containerName="main" containerID="cri-o://07695c9d8ee041361f82e87697f1d91aadcda1f17abb1f1579e0aaec57cad5ca" gracePeriod=45
Sep 18 16:26:07.740763 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.740709    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:26:07.844009 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.843979    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-model-cache\") pod \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") "
Sep 18 16:26:07.844161 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.844013    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-home\") pod \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") "
Sep 18 16:26:07.844161 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.844078    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-tmp-dir\") pod \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") "
Sep 18 16:26:07.844285 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.844173    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksvvd\" (UniqueName: \"kubernetes.io/projected/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-kube-api-access-ksvvd\") pod \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") "
Sep 18 16:26:07.844285 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.844213    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-dshm\") pod \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") "
Sep 18 16:26:07.844285 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.844245    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fe3ee55-fe3f-4939-989a-a2d460cf8c16" (UID: "5fe3ee55-fe3f-4939-989a-a2d460cf8c16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:26:07.844285 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.844274    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-tls-certs\") pod \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\" (UID: \"5fe3ee55-fe3f-4939-989a-a2d460cf8c16\") "
Sep 18 16:26:07.844464 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.844288    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-home" (OuterVolumeSpecName: "home") pod "5fe3ee55-fe3f-4939-989a-a2d460cf8c16" (UID: "5fe3ee55-fe3f-4939-989a-a2d460cf8c16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:26:07.844464 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.844381    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5fe3ee55-fe3f-4939-989a-a2d460cf8c16" (UID: "5fe3ee55-fe3f-4939-989a-a2d460cf8c16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:26:07.844600 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.844585    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:07.844654 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.844602    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:07.844654 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.844611    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:07.846416 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.846386    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-dshm" (OuterVolumeSpecName: "dshm") pod "5fe3ee55-fe3f-4939-989a-a2d460cf8c16" (UID: "5fe3ee55-fe3f-4939-989a-a2d460cf8c16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:26:07.846525 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.846451    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fe3ee55-fe3f-4939-989a-a2d460cf8c16" (UID: "5fe3ee55-fe3f-4939-989a-a2d460cf8c16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:26:07.846525 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.846459    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-kube-api-access-ksvvd" (OuterVolumeSpecName: "kube-api-access-ksvvd") pod "5fe3ee55-fe3f-4939-989a-a2d460cf8c16" (UID: "5fe3ee55-fe3f-4939-989a-a2d460cf8c16"). InnerVolumeSpecName "kube-api-access-ksvvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:26:07.945936 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.945911    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksvvd\" (UniqueName: \"kubernetes.io/projected/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-kube-api-access-ksvvd\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:07.945936 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.945933    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:07.946103 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:07.945943    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe3ee55-fe3f-4939-989a-a2d460cf8c16-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:08.166870 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:08.166794    2546 generic.go:356] "Generic (PLEG): container finished" podID="5fe3ee55-fe3f-4939-989a-a2d460cf8c16" containerID="07695c9d8ee041361f82e87697f1d91aadcda1f17abb1f1579e0aaec57cad5ca" exitCode=0
Sep 18 16:26:08.166870 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:08.166831    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"
Sep 18 16:26:08.167055 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:08.166877    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z" event={"ID":"5fe3ee55-fe3f-4939-989a-a2d460cf8c16","Type":"ContainerDied","Data":"07695c9d8ee041361f82e87697f1d91aadcda1f17abb1f1579e0aaec57cad5ca"}
Sep 18 16:26:08.167055 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:08.166917    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z" event={"ID":"5fe3ee55-fe3f-4939-989a-a2d460cf8c16","Type":"ContainerDied","Data":"798a1805894f95a30d458605de3752c6d9e7b13c917ab7979c3572fe4fd12220"}
Sep 18 16:26:08.167055 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:08.166935    2546 scope.go:117] "RemoveContainer" containerID="07695c9d8ee041361f82e87697f1d91aadcda1f17abb1f1579e0aaec57cad5ca"
Sep 18 16:26:08.176242 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:08.176228    2546 scope.go:117] "RemoveContainer" containerID="07695c9d8ee041361f82e87697f1d91aadcda1f17abb1f1579e0aaec57cad5ca"
Sep 18 16:26:08.176491 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:08.176472    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07695c9d8ee041361f82e87697f1d91aadcda1f17abb1f1579e0aaec57cad5ca\": container with ID starting with 07695c9d8ee041361f82e87697f1d91aadcda1f17abb1f1579e0aaec57cad5ca not found: ID does not exist" containerID="07695c9d8ee041361f82e87697f1d91aadcda1f17abb1f1579e0aaec57cad5ca"
Sep 18 16:26:08.176531 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:08.176498    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07695c9d8ee041361f82e87697f1d91aadcda1f17abb1f1579e0aaec57cad5ca"} err="failed to get container status \"07695c9d8ee041361f82e87697f1d91aadcda1f17abb1f1579e0aaec57cad5ca\": rpc error: code = NotFound desc = could not find container \"07695c9d8ee041361f82e87697f1d91aadcda1f17abb1f1579e0aaec57cad5ca\": container with ID starting with 07695c9d8ee041361f82e87697f1d91aadcda1f17abb1f1579e0aaec57cad5ca not found: ID does not exist"
Sep 18 16:26:08.191246 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:08.191224    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"]
Sep 18 16:26:08.201726 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:08.201707    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-72547f/noten-model-72547f-kserve-7c975f8dc-l4b6z"]
Sep 18 16:26:08.603187 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:08.603162    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe3ee55-fe3f-4939-989a-a2d460cf8c16" path="/var/lib/kubelet/pods/5fe3ee55-fe3f-4939-989a-a2d460cf8c16/volumes"
Sep 18 16:26:15.765726 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:15.765695    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:26:28.007774 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:28.007732    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-5b64c3"]
Sep 18 16:26:28.008443 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:28.008167    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe3ee55-fe3f-4939-989a-a2d460cf8c16" containerName="main"
Sep 18 16:26:28.008443 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:28.008179    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe3ee55-fe3f-4939-989a-a2d460cf8c16" containerName="main"
Sep 18 16:26:28.008443 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:28.008262    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe3ee55-fe3f-4939-989a-a2d460cf8c16" containerName="main"
Sep 18 16:26:28.013042 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:28.013006    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-5b64c3"
Sep 18 16:26:28.018723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:28.018701    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6q5gq\""
Sep 18 16:26:28.027061 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:28.027010    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-5b64c3"]
Sep 18 16:26:28.043711 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:28.043684    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqqd2\" (UniqueName: \"kubernetes.io/projected/86ea0077-e378-4288-bc46-fbf4419eec31-kube-api-access-fqqd2\") pod \"test-curl-7579-5b64c3\" (UID: \"86ea0077-e378-4288-bc46-fbf4419eec31\") " pod="opendatahub/test-curl-7579-5b64c3"
Sep 18 16:26:28.145300 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:28.145266    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqqd2\" (UniqueName: \"kubernetes.io/projected/86ea0077-e378-4288-bc46-fbf4419eec31-kube-api-access-fqqd2\") pod \"test-curl-7579-5b64c3\" (UID: \"86ea0077-e378-4288-bc46-fbf4419eec31\") " pod="opendatahub/test-curl-7579-5b64c3"
Sep 18 16:26:28.157618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:28.157593    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqqd2\" (UniqueName: \"kubernetes.io/projected/86ea0077-e378-4288-bc46-fbf4419eec31-kube-api-access-fqqd2\") pod \"test-curl-7579-5b64c3\" (UID: \"86ea0077-e378-4288-bc46-fbf4419eec31\") " pod="opendatahub/test-curl-7579-5b64c3"
Sep 18 16:26:28.322122 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:28.322044    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-5b64c3"
Sep 18 16:26:28.613907 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:28.613873    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-5b64c3"]
Sep 18 16:26:28.616617 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:26:28.616590    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86ea0077_e378_4288_bc46_fbf4419eec31.slice/crio-ce4098016c93755917752dd2f99a36b16eaae31ecab974a5c71f451e956fa0e2 WatchSource:0}: Error finding container ce4098016c93755917752dd2f99a36b16eaae31ecab974a5c71f451e956fa0e2: Status 404 returned error can't find the container with id ce4098016c93755917752dd2f99a36b16eaae31ecab974a5c71f451e956fa0e2
Sep 18 16:26:29.293486 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:29.293445    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-5b64c3" event={"ID":"86ea0077-e378-4288-bc46-fbf4419eec31","Type":"ContainerStarted","Data":"2ad93663bddb89a04a08a68f2fac824c406400b3416efdee55de1111e31f9e52"}
Sep 18 16:26:29.293861 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:29.293496    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-5b64c3" event={"ID":"86ea0077-e378-4288-bc46-fbf4419eec31","Type":"ContainerStarted","Data":"ce4098016c93755917752dd2f99a36b16eaae31ecab974a5c71f451e956fa0e2"}
Sep 18 16:26:29.312988 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:29.312942    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7579-5b64c3" podStartSLOduration=2.312925438 podStartE2EDuration="2.312925438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:29.311315911 +0000 UTC m=+1285.220096262" watchObservedRunningTime="2026-09-18 16:26:29.312925438 +0000 UTC m=+1285.221705789"
Sep 18 16:26:29.934756 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:29.934722    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-5b64c3"]
Sep 18 16:26:29.958363 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:29.958336    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-5b64c3"]
Sep 18 16:26:30.093684 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:30.093597    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-7f0ef0"]
Sep 18 16:26:30.094209 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:30.094192    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ea0077-e378-4288-bc46-fbf4419eec31" containerName="test-curl-7579-5b64c3"
Sep 18 16:26:30.094284 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:30.094211    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ea0077-e378-4288-bc46-fbf4419eec31" containerName="test-curl-7579-5b64c3"
Sep 18 16:26:30.094338 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:30.094316    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ea0077-e378-4288-bc46-fbf4419eec31" containerName="test-curl-7579-5b64c3"
Sep 18 16:26:30.097742 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:30.097724    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-7f0ef0"
Sep 18 16:26:30.109866 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:30.109842    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-7f0ef0"]
Sep 18 16:26:30.165940 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:30.165912    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqm52\" (UniqueName: \"kubernetes.io/projected/0f1ba5f5-b155-42e0-834f-597c796dd88f-kube-api-access-dqm52\") pod \"test-curl-7579-7f0ef0\" (UID: \"0f1ba5f5-b155-42e0-834f-597c796dd88f\") " pod="opendatahub/test-curl-7579-7f0ef0"
Sep 18 16:26:30.267538 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:30.267507    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqm52\" (UniqueName: \"kubernetes.io/projected/0f1ba5f5-b155-42e0-834f-597c796dd88f-kube-api-access-dqm52\") pod \"test-curl-7579-7f0ef0\" (UID: \"0f1ba5f5-b155-42e0-834f-597c796dd88f\") " pod="opendatahub/test-curl-7579-7f0ef0"
Sep 18 16:26:30.278541 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:30.278511    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqm52\" (UniqueName: \"kubernetes.io/projected/0f1ba5f5-b155-42e0-834f-597c796dd88f-kube-api-access-dqm52\") pod \"test-curl-7579-7f0ef0\" (UID: \"0f1ba5f5-b155-42e0-834f-597c796dd88f\") " pod="opendatahub/test-curl-7579-7f0ef0"
Sep 18 16:26:30.299455 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:30.299430    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7579-5b64c3" container="test-curl-7579-5b64c3"
Sep 18 16:26:30.406732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:30.406669    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-7f0ef0"
Sep 18 16:26:30.685767 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:30.685739    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-7f0ef0"]
Sep 18 16:26:30.687285 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:26:30.687255    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f1ba5f5_b155_42e0_834f_597c796dd88f.slice/crio-bf320a2cd6acfb7b508d7adb73cdc31dc780b646c158b4bd566067a164469c7a WatchSource:0}: Error finding container bf320a2cd6acfb7b508d7adb73cdc31dc780b646c158b4bd566067a164469c7a: Status 404 returned error can't find the container with id bf320a2cd6acfb7b508d7adb73cdc31dc780b646c158b4bd566067a164469c7a
Sep 18 16:26:31.306988 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:31.306957    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-7f0ef0" event={"ID":"0f1ba5f5-b155-42e0-834f-597c796dd88f","Type":"ContainerStarted","Data":"e1fe57297f7f4ab802ef9b1261af59fa78f53d565573dc9d0d2a363fe18f3ae8"}
Sep 18 16:26:31.307443 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:31.307000    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-7f0ef0" event={"ID":"0f1ba5f5-b155-42e0-834f-597c796dd88f","Type":"ContainerStarted","Data":"bf320a2cd6acfb7b508d7adb73cdc31dc780b646c158b4bd566067a164469c7a"}
Sep 18 16:26:31.307443 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:31.307133    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7579-5b64c3" podUID="86ea0077-e378-4288-bc46-fbf4419eec31" containerName="test-curl-7579-5b64c3" containerID="cri-o://2ad93663bddb89a04a08a68f2fac824c406400b3416efdee55de1111e31f9e52" gracePeriod=2
Sep 18 16:26:31.329471 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:31.329430    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7579-7f0ef0" podStartSLOduration=1.329413287 podStartE2EDuration="1.329413287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:31.326707885 +0000 UTC m=+1287.235488235" watchObservedRunningTime="2026-09-18 16:26:31.329413287 +0000 UTC m=+1287.238193640"
Sep 18 16:26:31.964653 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:31.964623    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-7f0ef0"]
Sep 18 16:26:31.972887 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:31.972863    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-7f0ef0"]
Sep 18 16:26:32.089369 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.089333    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-53b5d8"]
Sep 18 16:26:32.089920 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.089905    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1ba5f5-b155-42e0-834f-597c796dd88f" containerName="test-curl-7579-7f0ef0"
Sep 18 16:26:32.089982 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.089923    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1ba5f5-b155-42e0-834f-597c796dd88f" containerName="test-curl-7579-7f0ef0"
Sep 18 16:26:32.090062 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.090050    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1ba5f5-b155-42e0-834f-597c796dd88f" containerName="test-curl-7579-7f0ef0"
Sep 18 16:26:32.093268 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.093250    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-53b5d8"
Sep 18 16:26:32.115883 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.115856    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-53b5d8"]
Sep 18 16:26:32.188811 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.188780    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cshc8\" (UniqueName: \"kubernetes.io/projected/b9b29890-2801-457d-a12d-ce7d81034b0d-kube-api-access-cshc8\") pod \"test-curl-7579-53b5d8\" (UID: \"b9b29890-2801-457d-a12d-ce7d81034b0d\") " pod="opendatahub/test-curl-7579-53b5d8"
Sep 18 16:26:32.289604 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.289524    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cshc8\" (UniqueName: \"kubernetes.io/projected/b9b29890-2801-457d-a12d-ce7d81034b0d-kube-api-access-cshc8\") pod \"test-curl-7579-53b5d8\" (UID: \"b9b29890-2801-457d-a12d-ce7d81034b0d\") " pod="opendatahub/test-curl-7579-53b5d8"
Sep 18 16:26:32.304950 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.304925    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cshc8\" (UniqueName: \"kubernetes.io/projected/b9b29890-2801-457d-a12d-ce7d81034b0d-kube-api-access-cshc8\") pod \"test-curl-7579-53b5d8\" (UID: \"b9b29890-2801-457d-a12d-ce7d81034b0d\") " pod="opendatahub/test-curl-7579-53b5d8"
Sep 18 16:26:32.313538 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:32.313517    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7579-7f0ef0" container="test-curl-7579-7f0ef0"
Sep 18 16:26:32.349873 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.349847    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"]
Sep 18 16:26:32.353432 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.353415    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.357007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.356988    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1eec1d\"/\"distinct-w0-1eec1d-kserve-self-signed-certs\""
Sep 18 16:26:32.368092 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.368071    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"]
Sep 18 16:26:32.391198 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.391178    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-dshm\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.391300 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.391208    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljd4k\" (UniqueName: \"kubernetes.io/projected/4431196d-8458-4694-a9f5-6831131c6467-kube-api-access-ljd4k\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.391300 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.391277    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-model-cache\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.391397 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.391329    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4431196d-8458-4694-a9f5-6831131c6467-tls-certs\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.391432 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.391397    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-tmp-dir\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.391432 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.391419    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-home\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.401640 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.401616    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-53b5d8"
Sep 18 16:26:32.492617 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.492584    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-dshm\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.492785 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.492632    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljd4k\" (UniqueName: \"kubernetes.io/projected/4431196d-8458-4694-a9f5-6831131c6467-kube-api-access-ljd4k\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.492785 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.492719    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-model-cache\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.492934 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.492909    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4431196d-8458-4694-a9f5-6831131c6467-tls-certs\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.493096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.493066    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-tmp-dir\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.493213 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.493121    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-model-cache\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.493213 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.493142    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-home\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.493360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.493341    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-tmp-dir\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.493525 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.493503    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-home\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.494864 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.494842    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-dshm\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.495453 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.495421    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4431196d-8458-4694-a9f5-6831131c6467-tls-certs\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.504918 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.504895    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljd4k\" (UniqueName: \"kubernetes.io/projected/4431196d-8458-4694-a9f5-6831131c6467-kube-api-access-ljd4k\") pod \"distinct-w0-1eec1d-kserve-77988877b-zz28j\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.663275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.663204    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:32.675936 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.675919    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-53b5d8"]
Sep 18 16:26:32.677616 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:26:32.677582    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9b29890_2801_457d_a12d_ce7d81034b0d.slice/crio-1a3cae8e68f069034d6ecbfd30d00a25cb5d6c7fd29ea9fcfdae57202ab78225 WatchSource:0}: Error finding container 1a3cae8e68f069034d6ecbfd30d00a25cb5d6c7fd29ea9fcfdae57202ab78225: Status 404 returned error can't find the container with id 1a3cae8e68f069034d6ecbfd30d00a25cb5d6c7fd29ea9fcfdae57202ab78225
Sep 18 16:26:32.952816 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:32.952791    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"]
Sep 18 16:26:32.953660 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:26:32.953634    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4431196d_8458_4694_a9f5_6831131c6467.slice/crio-8ff65739d91ac2aadcc84efccb23a3cdd1932acc3ff9791058ea62e22eb423af WatchSource:0}: Error finding container 8ff65739d91ac2aadcc84efccb23a3cdd1932acc3ff9791058ea62e22eb423af: Status 404 returned error can't find the container with id 8ff65739d91ac2aadcc84efccb23a3cdd1932acc3ff9791058ea62e22eb423af
Sep 18 16:26:33.322014 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:33.321979    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-53b5d8" event={"ID":"b9b29890-2801-457d-a12d-ce7d81034b0d","Type":"ContainerStarted","Data":"78ba7b2e272968419e385e7ccd754cc0135db07b2b369a26c841168a32026933"}
Sep 18 16:26:33.322401 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:33.322061    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-53b5d8" event={"ID":"b9b29890-2801-457d-a12d-ce7d81034b0d","Type":"ContainerStarted","Data":"1a3cae8e68f069034d6ecbfd30d00a25cb5d6c7fd29ea9fcfdae57202ab78225"}
Sep 18 16:26:33.323608 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:33.323583    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j" event={"ID":"4431196d-8458-4694-a9f5-6831131c6467","Type":"ContainerStarted","Data":"8eadd26e5eb2221f892091f1d620980a24fdd33f750bbc11ffabdc17562367a9"}
Sep 18 16:26:33.323714 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:33.323624    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j" event={"ID":"4431196d-8458-4694-a9f5-6831131c6467","Type":"ContainerStarted","Data":"8ff65739d91ac2aadcc84efccb23a3cdd1932acc3ff9791058ea62e22eb423af"}
Sep 18 16:26:33.323714 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:33.323655    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7579-7f0ef0" podUID="0f1ba5f5-b155-42e0-834f-597c796dd88f" containerName="test-curl-7579-7f0ef0" containerID="cri-o://e1fe57297f7f4ab802ef9b1261af59fa78f53d565573dc9d0d2a363fe18f3ae8" gracePeriod=2
Sep 18 16:26:33.343590 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:33.343553    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7579-53b5d8" podStartSLOduration=1.34354005 podStartE2EDuration="1.34354005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:33.340589331 +0000 UTC m=+1289.249369680" watchObservedRunningTime="2026-09-18 16:26:33.34354005 +0000 UTC m=+1289.252320400"
Sep 18 16:26:33.367607 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:33.367562    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j" podStartSLOduration=1.367548355 podStartE2EDuration="1.367548355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:33.364077019 +0000 UTC m=+1289.272857388" watchObservedRunningTime="2026-09-18 16:26:33.367548355 +0000 UTC m=+1289.276328759"
Sep 18 16:26:33.460418 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:33.460386    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-5b64c3"
Sep 18 16:26:33.503468 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:33.503439    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqqd2\" (UniqueName: \"kubernetes.io/projected/86ea0077-e378-4288-bc46-fbf4419eec31-kube-api-access-fqqd2\") pod \"86ea0077-e378-4288-bc46-fbf4419eec31\" (UID: \"86ea0077-e378-4288-bc46-fbf4419eec31\") "
Sep 18 16:26:33.505434 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:33.505405    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ea0077-e378-4288-bc46-fbf4419eec31-kube-api-access-fqqd2" (OuterVolumeSpecName: "kube-api-access-fqqd2") pod "86ea0077-e378-4288-bc46-fbf4419eec31" (UID: "86ea0077-e378-4288-bc46-fbf4419eec31"). InnerVolumeSpecName "kube-api-access-fqqd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:26:33.604466 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:33.604389    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqqd2\" (UniqueName: \"kubernetes.io/projected/86ea0077-e378-4288-bc46-fbf4419eec31-kube-api-access-fqqd2\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:33.956641 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:33.956603    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-53b5d8"]
Sep 18 16:26:33.959316 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:33.959286    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-53b5d8"]
Sep 18 16:26:34.078698 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.078596    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7579-93e092"]
Sep 18 16:26:34.079302 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.079276    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b29890-2801-457d-a12d-ce7d81034b0d" containerName="test-curl-7579-53b5d8"
Sep 18 16:26:34.079302 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.079301    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b29890-2801-457d-a12d-ce7d81034b0d" containerName="test-curl-7579-53b5d8"
Sep 18 16:26:34.079541 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.079394    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b29890-2801-457d-a12d-ce7d81034b0d" containerName="test-curl-7579-53b5d8"
Sep 18 16:26:34.082649 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.082626    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-93e092"
Sep 18 16:26:34.098409 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.098384    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-93e092"]
Sep 18 16:26:34.109662 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.109634    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cjg4\" (UniqueName: \"kubernetes.io/projected/4a69b7bd-bd50-4188-8548-380f588cecdc-kube-api-access-6cjg4\") pod \"test-curl-7579-93e092\" (UID: \"4a69b7bd-bd50-4188-8548-380f588cecdc\") " pod="opendatahub/test-curl-7579-93e092"
Sep 18 16:26:34.210567 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.210498    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cjg4\" (UniqueName: \"kubernetes.io/projected/4a69b7bd-bd50-4188-8548-380f588cecdc-kube-api-access-6cjg4\") pod \"test-curl-7579-93e092\" (UID: \"4a69b7bd-bd50-4188-8548-380f588cecdc\") " pod="opendatahub/test-curl-7579-93e092"
Sep 18 16:26:34.220955 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.220931    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cjg4\" (UniqueName: \"kubernetes.io/projected/4a69b7bd-bd50-4188-8548-380f588cecdc-kube-api-access-6cjg4\") pod \"test-curl-7579-93e092\" (UID: \"4a69b7bd-bd50-4188-8548-380f588cecdc\") " pod="opendatahub/test-curl-7579-93e092"
Sep 18 16:26:34.329855 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.329826    2546 generic.go:356] "Generic (PLEG): container finished" podID="86ea0077-e378-4288-bc46-fbf4419eec31" containerID="2ad93663bddb89a04a08a68f2fac824c406400b3416efdee55de1111e31f9e52" exitCode=137
Sep 18 16:26:34.330279 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.329862    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-5b64c3"
Sep 18 16:26:34.330279 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.329935    2546 scope.go:117] "RemoveContainer" containerID="2ad93663bddb89a04a08a68f2fac824c406400b3416efdee55de1111e31f9e52"
Sep 18 16:26:34.330279 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:34.330218    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7579-53b5d8" container="test-curl-7579-53b5d8"
Sep 18 16:26:34.338919 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.338905    2546 scope.go:117] "RemoveContainer" containerID="2ad93663bddb89a04a08a68f2fac824c406400b3416efdee55de1111e31f9e52"
Sep 18 16:26:34.339183 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:34.339166    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad93663bddb89a04a08a68f2fac824c406400b3416efdee55de1111e31f9e52\": container with ID starting with 2ad93663bddb89a04a08a68f2fac824c406400b3416efdee55de1111e31f9e52 not found: ID does not exist" containerID="2ad93663bddb89a04a08a68f2fac824c406400b3416efdee55de1111e31f9e52"
Sep 18 16:26:34.339256 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.339190    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad93663bddb89a04a08a68f2fac824c406400b3416efdee55de1111e31f9e52"} err="failed to get container status \"2ad93663bddb89a04a08a68f2fac824c406400b3416efdee55de1111e31f9e52\": rpc error: code = NotFound desc = could not find container \"2ad93663bddb89a04a08a68f2fac824c406400b3416efdee55de1111e31f9e52\": container with ID starting with 2ad93663bddb89a04a08a68f2fac824c406400b3416efdee55de1111e31f9e52 not found: ID does not exist"
Sep 18 16:26:34.341246 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:34.341227    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7579-5b64c3\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="86ea0077-e378-4288-bc46-fbf4419eec31" pod="opendatahub/test-curl-7579-5b64c3"
Sep 18 16:26:34.392638 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.392614    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-93e092"
Sep 18 16:26:34.603364 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:34.603284    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7579-5b64c3\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="86ea0077-e378-4288-bc46-fbf4419eec31" pod="opendatahub/test-curl-7579-5b64c3"
Sep 18 16:26:34.605461 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.605434    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86ea0077-e378-4288-bc46-fbf4419eec31" path="/var/lib/kubelet/pods/86ea0077-e378-4288-bc46-fbf4419eec31/volumes"
Sep 18 16:26:34.674384 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:26:34.674352    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a69b7bd_bd50_4188_8548_380f588cecdc.slice/crio-d0e87e373a63ec1574602b24acaaae571a1085959dc3847f8d9dc72e97f15317 WatchSource:0}: Error finding container d0e87e373a63ec1574602b24acaaae571a1085959dc3847f8d9dc72e97f15317: Status 404 returned error can't find the container with id d0e87e373a63ec1574602b24acaaae571a1085959dc3847f8d9dc72e97f15317
Sep 18 16:26:34.674508 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:34.674455    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7579-93e092"]
Sep 18 16:26:35.120436 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:35.120407    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"]
Sep 18 16:26:35.338584 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:35.338398    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-93e092" event={"ID":"4a69b7bd-bd50-4188-8548-380f588cecdc","Type":"ContainerStarted","Data":"388238cdba17623c9d86720e2d2cc1104749b0779ccaf245c9f8773c62858229"}
Sep 18 16:26:35.338584 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:35.338436    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7579-93e092" event={"ID":"4a69b7bd-bd50-4188-8548-380f588cecdc","Type":"ContainerStarted","Data":"d0e87e373a63ec1574602b24acaaae571a1085959dc3847f8d9dc72e97f15317"}
Sep 18 16:26:35.338584 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:35.338497    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7579-53b5d8" podUID="b9b29890-2801-457d-a12d-ce7d81034b0d" containerName="test-curl-7579-53b5d8" containerID="cri-o://78ba7b2e272968419e385e7ccd754cc0135db07b2b369a26c841168a32026933" gracePeriod=2
Sep 18 16:26:35.354807 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:35.354764    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7579-93e092" podStartSLOduration=1.354749899 podStartE2EDuration="1.354749899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:35.352764313 +0000 UTC m=+1291.261544663" watchObservedRunningTime="2026-09-18 16:26:35.354749899 +0000 UTC m=+1291.263530303"
Sep 18 16:26:35.511446 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:35.511423    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-7f0ef0"
Sep 18 16:26:35.627951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:35.627927    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqm52\" (UniqueName: \"kubernetes.io/projected/0f1ba5f5-b155-42e0-834f-597c796dd88f-kube-api-access-dqm52\") pod \"0f1ba5f5-b155-42e0-834f-597c796dd88f\" (UID: \"0f1ba5f5-b155-42e0-834f-597c796dd88f\") "
Sep 18 16:26:35.629807 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:35.629775    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1ba5f5-b155-42e0-834f-597c796dd88f-kube-api-access-dqm52" (OuterVolumeSpecName: "kube-api-access-dqm52") pod "0f1ba5f5-b155-42e0-834f-597c796dd88f" (UID: "0f1ba5f5-b155-42e0-834f-597c796dd88f"). InnerVolumeSpecName "kube-api-access-dqm52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:26:35.729160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:35.729136    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqm52\" (UniqueName: \"kubernetes.io/projected/0f1ba5f5-b155-42e0-834f-597c796dd88f-kube-api-access-dqm52\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:35.956365 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:35.956336    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7579-93e092"]
Sep 18 16:26:35.961522 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:35.961498    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7579-93e092"]
Sep 18 16:26:36.344904 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:36.344824    2546 generic.go:356] "Generic (PLEG): container finished" podID="0f1ba5f5-b155-42e0-834f-597c796dd88f" containerID="e1fe57297f7f4ab802ef9b1261af59fa78f53d565573dc9d0d2a363fe18f3ae8" exitCode=137
Sep 18 16:26:36.344904 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:36.344862    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-7f0ef0"
Sep 18 16:26:36.345318 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:36.344907    2546 scope.go:117] "RemoveContainer" containerID="e1fe57297f7f4ab802ef9b1261af59fa78f53d565573dc9d0d2a363fe18f3ae8"
Sep 18 16:26:36.345318 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:36.345240    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7579-93e092" container="test-curl-7579-93e092"
Sep 18 16:26:36.356746 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:36.356728    2546 scope.go:117] "RemoveContainer" containerID="e1fe57297f7f4ab802ef9b1261af59fa78f53d565573dc9d0d2a363fe18f3ae8"
Sep 18 16:26:36.356994 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:36.356977    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fe57297f7f4ab802ef9b1261af59fa78f53d565573dc9d0d2a363fe18f3ae8\": container with ID starting with e1fe57297f7f4ab802ef9b1261af59fa78f53d565573dc9d0d2a363fe18f3ae8 not found: ID does not exist" containerID="e1fe57297f7f4ab802ef9b1261af59fa78f53d565573dc9d0d2a363fe18f3ae8"
Sep 18 16:26:36.357050 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:36.356999    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fe57297f7f4ab802ef9b1261af59fa78f53d565573dc9d0d2a363fe18f3ae8"} err="failed to get container status \"e1fe57297f7f4ab802ef9b1261af59fa78f53d565573dc9d0d2a363fe18f3ae8\": rpc error: code = NotFound desc = could not find container \"e1fe57297f7f4ab802ef9b1261af59fa78f53d565573dc9d0d2a363fe18f3ae8\": container with ID starting with e1fe57297f7f4ab802ef9b1261af59fa78f53d565573dc9d0d2a363fe18f3ae8 not found: ID does not exist"
Sep 18 16:26:36.358167 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:36.358145    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7579-7f0ef0\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="0f1ba5f5-b155-42e0-834f-597c796dd88f" pod="opendatahub/test-curl-7579-7f0ef0"
Sep 18 16:26:36.603225 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:36.603159    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f1ba5f5-b155-42e0-834f-597c796dd88f" path="/var/lib/kubelet/pods/0f1ba5f5-b155-42e0-834f-597c796dd88f/volumes"
Sep 18 16:26:37.351834 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:37.351807    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7579-93e092" podUID="4a69b7bd-bd50-4188-8548-380f588cecdc" containerName="test-curl-7579-93e092" containerID="cri-o://388238cdba17623c9d86720e2d2cc1104749b0779ccaf245c9f8773c62858229" gracePeriod=2
Sep 18 16:26:37.524247 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:37.524223    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-53b5d8"
Sep 18 16:26:37.648808 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:37.648727    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cshc8\" (UniqueName: \"kubernetes.io/projected/b9b29890-2801-457d-a12d-ce7d81034b0d-kube-api-access-cshc8\") pod \"b9b29890-2801-457d-a12d-ce7d81034b0d\" (UID: \"b9b29890-2801-457d-a12d-ce7d81034b0d\") "
Sep 18 16:26:37.650916 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:37.650886    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b29890-2801-457d-a12d-ce7d81034b0d-kube-api-access-cshc8" (OuterVolumeSpecName: "kube-api-access-cshc8") pod "b9b29890-2801-457d-a12d-ce7d81034b0d" (UID: "b9b29890-2801-457d-a12d-ce7d81034b0d"). InnerVolumeSpecName "kube-api-access-cshc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:26:37.750225 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:37.750200    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cshc8\" (UniqueName: \"kubernetes.io/projected/b9b29890-2801-457d-a12d-ce7d81034b0d-kube-api-access-cshc8\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:38.358250 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:38.358219    2546 generic.go:356] "Generic (PLEG): container finished" podID="b9b29890-2801-457d-a12d-ce7d81034b0d" containerID="78ba7b2e272968419e385e7ccd754cc0135db07b2b369a26c841168a32026933" exitCode=137
Sep 18 16:26:38.358701 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:38.358267    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-53b5d8"
Sep 18 16:26:38.358701 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:38.358315    2546 scope.go:117] "RemoveContainer" containerID="78ba7b2e272968419e385e7ccd754cc0135db07b2b369a26c841168a32026933"
Sep 18 16:26:38.367196 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:38.367171    2546 scope.go:117] "RemoveContainer" containerID="78ba7b2e272968419e385e7ccd754cc0135db07b2b369a26c841168a32026933"
Sep 18 16:26:38.367445 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:38.367428    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ba7b2e272968419e385e7ccd754cc0135db07b2b369a26c841168a32026933\": container with ID starting with 78ba7b2e272968419e385e7ccd754cc0135db07b2b369a26c841168a32026933 not found: ID does not exist" containerID="78ba7b2e272968419e385e7ccd754cc0135db07b2b369a26c841168a32026933"
Sep 18 16:26:38.367501 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:38.367455    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ba7b2e272968419e385e7ccd754cc0135db07b2b369a26c841168a32026933"} err="failed to get container status \"78ba7b2e272968419e385e7ccd754cc0135db07b2b369a26c841168a32026933\": rpc error: code = NotFound desc = could not find container \"78ba7b2e272968419e385e7ccd754cc0135db07b2b369a26c841168a32026933\": container with ID starting with 78ba7b2e272968419e385e7ccd754cc0135db07b2b369a26c841168a32026933 not found: ID does not exist"
Sep 18 16:26:38.368997 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:38.368964    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7579-53b5d8\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="b9b29890-2801-457d-a12d-ce7d81034b0d" pod="opendatahub/test-curl-7579-53b5d8"
Sep 18 16:26:38.607110 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:38.606983    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b29890-2801-457d-a12d-ce7d81034b0d" path="/var/lib/kubelet/pods/b9b29890-2801-457d-a12d-ce7d81034b0d/volumes"
Sep 18 16:26:39.524931 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:39.524906    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-93e092"
Sep 18 16:26:39.672405 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:39.672327    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cjg4\" (UniqueName: \"kubernetes.io/projected/4a69b7bd-bd50-4188-8548-380f588cecdc-kube-api-access-6cjg4\") pod \"4a69b7bd-bd50-4188-8548-380f588cecdc\" (UID: \"4a69b7bd-bd50-4188-8548-380f588cecdc\") "
Sep 18 16:26:39.674338 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:39.674316    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a69b7bd-bd50-4188-8548-380f588cecdc-kube-api-access-6cjg4" (OuterVolumeSpecName: "kube-api-access-6cjg4") pod "4a69b7bd-bd50-4188-8548-380f588cecdc" (UID: "4a69b7bd-bd50-4188-8548-380f588cecdc"). InnerVolumeSpecName "kube-api-access-6cjg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:26:39.773688 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:39.773663    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cjg4\" (UniqueName: \"kubernetes.io/projected/4a69b7bd-bd50-4188-8548-380f588cecdc-kube-api-access-6cjg4\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:39.993686 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:39.993642    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:26:40.373440 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:40.373357    2546 generic.go:356] "Generic (PLEG): container finished" podID="4a69b7bd-bd50-4188-8548-380f588cecdc" containerID="388238cdba17623c9d86720e2d2cc1104749b0779ccaf245c9f8773c62858229" exitCode=137
Sep 18 16:26:40.373440 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:40.373398    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7579-93e092"
Sep 18 16:26:40.373636 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:40.373472    2546 scope.go:117] "RemoveContainer" containerID="388238cdba17623c9d86720e2d2cc1104749b0779ccaf245c9f8773c62858229"
Sep 18 16:26:40.384406 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:40.384384    2546 scope.go:117] "RemoveContainer" containerID="388238cdba17623c9d86720e2d2cc1104749b0779ccaf245c9f8773c62858229"
Sep 18 16:26:40.384722 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:40.384703    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388238cdba17623c9d86720e2d2cc1104749b0779ccaf245c9f8773c62858229\": container with ID starting with 388238cdba17623c9d86720e2d2cc1104749b0779ccaf245c9f8773c62858229 not found: ID does not exist" containerID="388238cdba17623c9d86720e2d2cc1104749b0779ccaf245c9f8773c62858229"
Sep 18 16:26:40.384828 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:40.384728    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388238cdba17623c9d86720e2d2cc1104749b0779ccaf245c9f8773c62858229"} err="failed to get container status \"388238cdba17623c9d86720e2d2cc1104749b0779ccaf245c9f8773c62858229\": rpc error: code = NotFound desc = could not find container \"388238cdba17623c9d86720e2d2cc1104749b0779ccaf245c9f8773c62858229\": container with ID starting with 388238cdba17623c9d86720e2d2cc1104749b0779ccaf245c9f8773c62858229 not found: ID does not exist"
Sep 18 16:26:40.386125 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:40.386102    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7579-93e092\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="4a69b7bd-bd50-4188-8548-380f588cecdc" pod="opendatahub/test-curl-7579-93e092"
Sep 18 16:26:40.603454 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:40.603421    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a69b7bd-bd50-4188-8548-380f588cecdc" path="/var/lib/kubelet/pods/4a69b7bd-bd50-4188-8548-380f588cecdc/volumes"
Sep 18 16:26:42.663374 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:42.663343    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:42.663815 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:42.663388    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:42.675920 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:42.675892    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:43.405360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:43.405331    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:26:46.007581 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:46.007536    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58"]
Sep 18 16:26:46.007975 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:46.007959    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a69b7bd-bd50-4188-8548-380f588cecdc" containerName="test-curl-7579-93e092"
Sep 18 16:26:46.008020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:46.007978    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a69b7bd-bd50-4188-8548-380f588cecdc" containerName="test-curl-7579-93e092"
Sep 18 16:26:46.008096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:46.008085    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a69b7bd-bd50-4188-8548-380f588cecdc" containerName="test-curl-7579-93e092"
Sep 18 16:26:46.012402 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:46.012387    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58"
Sep 18 16:26:46.015312 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:46.015290    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:26:46.015609 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:46.015591    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:26:46.022712 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:46.022688    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58"]
Sep 18 16:26:46.143971 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:46.143939    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60440cdb-b821-4417-a3a4-3c713963b4f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58\" (UID: \"60440cdb-b821-4417-a3a4-3c713963b4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58"
Sep 18 16:26:46.245546 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:46.245518    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60440cdb-b821-4417-a3a4-3c713963b4f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58\" (UID: \"60440cdb-b821-4417-a3a4-3c713963b4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58"
Sep 18 16:26:46.246096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:46.246077    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60440cdb-b821-4417-a3a4-3c713963b4f8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58\" (UID: \"60440cdb-b821-4417-a3a4-3c713963b4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58"
Sep 18 16:26:46.323428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:46.323368    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58"
Sep 18 16:26:46.616896 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:46.616864    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58"]
Sep 18 16:26:46.618573 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:26:46.618545    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60440cdb_b821_4417_a3a4_3c713963b4f8.slice/crio-e80aaa6a1c691d5956b4b128ab07d57cb20e58d0d0a6ab4715954f9c5d47c5aa WatchSource:0}: Error finding container e80aaa6a1c691d5956b4b128ab07d57cb20e58d0d0a6ab4715954f9c5d47c5aa: Status 404 returned error can't find the container with id e80aaa6a1c691d5956b4b128ab07d57cb20e58d0d0a6ab4715954f9c5d47c5aa
Sep 18 16:26:47.418820 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:47.418728    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58" event={"ID":"60440cdb-b821-4417-a3a4-3c713963b4f8","Type":"ContainerStarted","Data":"81d3dd337536f89046edcd3845747d48e4879c30b8b780910b11f9a47da71eca"}
Sep 18 16:26:47.418820 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:47.418787    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58" event={"ID":"60440cdb-b821-4417-a3a4-3c713963b4f8","Type":"ContainerStarted","Data":"e80aaa6a1c691d5956b4b128ab07d57cb20e58d0d0a6ab4715954f9c5d47c5aa"}
Sep 18 16:26:47.444104 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:47.444055    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58" podStartSLOduration=2.444036588 podStartE2EDuration="2.444036588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:26:47.442050941 +0000 UTC m=+1303.350831307" watchObservedRunningTime="2026-09-18 16:26:47.444036588 +0000 UTC m=+1303.352816937"
Sep 18 16:26:48.426668 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:48.426634    2546 generic.go:356] "Generic (PLEG): container finished" podID="60440cdb-b821-4417-a3a4-3c713963b4f8" containerID="81d3dd337536f89046edcd3845747d48e4879c30b8b780910b11f9a47da71eca" exitCode=0
Sep 18 16:26:48.427085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:48.426743    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58" event={"ID":"60440cdb-b821-4417-a3a4-3c713963b4f8","Type":"ContainerDied","Data":"81d3dd337536f89046edcd3845747d48e4879c30b8b780910b11f9a47da71eca"}
Sep 18 16:26:49.567182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:49.567161    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58"
Sep 18 16:26:49.682809 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:49.682709    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60440cdb-b821-4417-a3a4-3c713963b4f8-maas-api-service-ca\") pod \"60440cdb-b821-4417-a3a4-3c713963b4f8\" (UID: \"60440cdb-b821-4417-a3a4-3c713963b4f8\") "
Sep 18 16:26:49.683057 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:49.683011    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60440cdb-b821-4417-a3a4-3c713963b4f8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60440cdb-b821-4417-a3a4-3c713963b4f8" (UID: "60440cdb-b821-4417-a3a4-3c713963b4f8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:26:49.683297 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:49.683278    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60440cdb-b821-4417-a3a4-3c713963b4f8-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:50.139788 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.139730    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns" podUID="309ae6b9-331c-46c1-9936-9e22dfa20dd9" containerName="main" containerID="cri-o://ff53d9e51b884738447cd73644f3306d0fd0cd09d26b816b80a404ac9acf3316" gracePeriod=45
Sep 18 16:26:50.406189 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.406132    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:26:50.439322 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.439297    2546 generic.go:356] "Generic (PLEG): container finished" podID="309ae6b9-331c-46c1-9936-9e22dfa20dd9" containerID="ff53d9e51b884738447cd73644f3306d0fd0cd09d26b816b80a404ac9acf3316" exitCode=0
Sep 18 16:26:50.439436 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.439362    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"
Sep 18 16:26:50.439517 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.439466    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns" event={"ID":"309ae6b9-331c-46c1-9936-9e22dfa20dd9","Type":"ContainerDied","Data":"ff53d9e51b884738447cd73644f3306d0fd0cd09d26b816b80a404ac9acf3316"}
Sep 18 16:26:50.439517 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.439505    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns" event={"ID":"309ae6b9-331c-46c1-9936-9e22dfa20dd9","Type":"ContainerDied","Data":"ace0fd3c3de3735d75613b3b10276b190834ed048fb89e9d9e28ba15da066aea"}
Sep 18 16:26:50.439623 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.439527    2546 scope.go:117] "RemoveContainer" containerID="ff53d9e51b884738447cd73644f3306d0fd0cd09d26b816b80a404ac9acf3316"
Sep 18 16:26:50.441342 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.441275    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58"
Sep 18 16:26:50.441342 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.441280    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58" event={"ID":"60440cdb-b821-4417-a3a4-3c713963b4f8","Type":"ContainerDied","Data":"e80aaa6a1c691d5956b4b128ab07d57cb20e58d0d0a6ab4715954f9c5d47c5aa"}
Sep 18 16:26:50.441342 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.441325    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e80aaa6a1c691d5956b4b128ab07d57cb20e58d0d0a6ab4715954f9c5d47c5aa"
Sep 18 16:26:50.450744 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.450723    2546 scope.go:117] "RemoveContainer" containerID="ff53d9e51b884738447cd73644f3306d0fd0cd09d26b816b80a404ac9acf3316"
Sep 18 16:26:50.451110 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:26:50.451082    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff53d9e51b884738447cd73644f3306d0fd0cd09d26b816b80a404ac9acf3316\": container with ID starting with ff53d9e51b884738447cd73644f3306d0fd0cd09d26b816b80a404ac9acf3316 not found: ID does not exist" containerID="ff53d9e51b884738447cd73644f3306d0fd0cd09d26b816b80a404ac9acf3316"
Sep 18 16:26:50.451209 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.451117    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff53d9e51b884738447cd73644f3306d0fd0cd09d26b816b80a404ac9acf3316"} err="failed to get container status \"ff53d9e51b884738447cd73644f3306d0fd0cd09d26b816b80a404ac9acf3316\": rpc error: code = NotFound desc = could not find container \"ff53d9e51b884738447cd73644f3306d0fd0cd09d26b816b80a404ac9acf3316\": container with ID starting with ff53d9e51b884738447cd73644f3306d0fd0cd09d26b816b80a404ac9acf3316 not found: ID does not exist"
Sep 18 16:26:50.491575 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.491555    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-model-cache\") pod \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") "
Sep 18 16:26:50.491676 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.491634    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309ae6b9-331c-46c1-9936-9e22dfa20dd9-tls-certs\") pod \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") "
Sep 18 16:26:50.491725 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.491716    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-tmp-dir\") pod \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") "
Sep 18 16:26:50.491802 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.491788    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-dshm\") pod \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") "
Sep 18 16:26:50.491850 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.491799    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-model-cache" (OuterVolumeSpecName: "model-cache") pod "309ae6b9-331c-46c1-9936-9e22dfa20dd9" (UID: "309ae6b9-331c-46c1-9936-9e22dfa20dd9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:26:50.491850 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.491816    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfgdw\" (UniqueName: \"kubernetes.io/projected/309ae6b9-331c-46c1-9936-9e22dfa20dd9-kube-api-access-qfgdw\") pod \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") "
Sep 18 16:26:50.491850 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.491842    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-home\") pod \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\" (UID: \"309ae6b9-331c-46c1-9936-9e22dfa20dd9\") "
Sep 18 16:26:50.491952 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.491911    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "309ae6b9-331c-46c1-9936-9e22dfa20dd9" (UID: "309ae6b9-331c-46c1-9936-9e22dfa20dd9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:26:50.492183 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.492157    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-home" (OuterVolumeSpecName: "home") pod "309ae6b9-331c-46c1-9936-9e22dfa20dd9" (UID: "309ae6b9-331c-46c1-9936-9e22dfa20dd9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:26:50.492289 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.492257    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:50.492346 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.492288    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:50.492346 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.492304    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:50.493777 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.493750    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309ae6b9-331c-46c1-9936-9e22dfa20dd9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "309ae6b9-331c-46c1-9936-9e22dfa20dd9" (UID: "309ae6b9-331c-46c1-9936-9e22dfa20dd9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:26:50.493854 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.493776    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-dshm" (OuterVolumeSpecName: "dshm") pod "309ae6b9-331c-46c1-9936-9e22dfa20dd9" (UID: "309ae6b9-331c-46c1-9936-9e22dfa20dd9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:26:50.493854 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.493802    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309ae6b9-331c-46c1-9936-9e22dfa20dd9-kube-api-access-qfgdw" (OuterVolumeSpecName: "kube-api-access-qfgdw") pod "309ae6b9-331c-46c1-9936-9e22dfa20dd9" (UID: "309ae6b9-331c-46c1-9936-9e22dfa20dd9"). InnerVolumeSpecName "kube-api-access-qfgdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:26:50.593443 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.593416    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309ae6b9-331c-46c1-9936-9e22dfa20dd9-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:50.593443 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.593442    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309ae6b9-331c-46c1-9936-9e22dfa20dd9-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:50.593749 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.593451    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfgdw\" (UniqueName: \"kubernetes.io/projected/309ae6b9-331c-46c1-9936-9e22dfa20dd9-kube-api-access-qfgdw\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:26:50.757808 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.757779    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"]
Sep 18 16:26:50.763181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:50.763157    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fbc529/test-model-fbc529-kserve-59579bccbc-5rdns"]
Sep 18 16:26:52.603019 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:52.602987    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="309ae6b9-331c-46c1-9936-9e22dfa20dd9" path="/var/lib/kubelet/pods/309ae6b9-331c-46c1-9936-9e22dfa20dd9/volumes"
Sep 18 16:26:57.373604 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:57.373569    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58"]
Sep 18 16:26:57.378819 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:57.377204    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fbc529-t7g58"]
Sep 18 16:26:58.603470 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:26:58.603432    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60440cdb-b821-4417-a3a4-3c713963b4f8" path="/var/lib/kubelet/pods/60440cdb-b821-4417-a3a4-3c713963b4f8/volumes"
Sep 18 16:27:04.785010 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:04.784975    2546 scope.go:117] "RemoveContainer" containerID="36a9aeb3762007fc692b793cc47bb4b7d9a1fdaf750ea546e347750e6113e433"
Sep 18 16:27:05.175819 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.175734    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"]
Sep 18 16:27:05.176039 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.175994    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy" containerID="cri-o://bb6a62dcf07783f0e3dabafabb53568d789f4ef1690887f54fb8dcd17743a2b9" gracePeriod=30
Sep 18 16:27:05.540228 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.540199    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x"]
Sep 18 16:27:05.540594 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.540582    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="309ae6b9-331c-46c1-9936-9e22dfa20dd9" containerName="main"
Sep 18 16:27:05.540650 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.540595    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="309ae6b9-331c-46c1-9936-9e22dfa20dd9" containerName="main"
Sep 18 16:27:05.540650 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.540606    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60440cdb-b821-4417-a3a4-3c713963b4f8" containerName="revoke-keys"
Sep 18 16:27:05.540650 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.540612    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="60440cdb-b821-4417-a3a4-3c713963b4f8" containerName="revoke-keys"
Sep 18 16:27:05.540751 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.540689    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="60440cdb-b821-4417-a3a4-3c713963b4f8" containerName="revoke-keys"
Sep 18 16:27:05.540751 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.540697    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="309ae6b9-331c-46c1-9936-9e22dfa20dd9" containerName="main"
Sep 18 16:27:05.545122 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.545104    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x"
Sep 18 16:27:05.548716 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.548460    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:27:05.548716 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.548461    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:27:05.583041 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.582191    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x"]
Sep 18 16:27:05.652017 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.651983    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b4ba4c-8f90-4ef2-996d-5883e6019ce6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x\" (UID: \"51b4ba4c-8f90-4ef2-996d-5883e6019ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x"
Sep 18 16:27:05.752897 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.752870    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b4ba4c-8f90-4ef2-996d-5883e6019ce6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x\" (UID: \"51b4ba4c-8f90-4ef2-996d-5883e6019ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x"
Sep 18 16:27:05.753410 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.753392    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b4ba4c-8f90-4ef2-996d-5883e6019ce6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x\" (UID: \"51b4ba4c-8f90-4ef2-996d-5883e6019ce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x"
Sep 18 16:27:05.857095 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:05.857038    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x"
Sep 18 16:27:06.141051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:06.141005    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x"]
Sep 18 16:27:06.141400 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:27:06.141370    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b4ba4c_8f90_4ef2_996d_5883e6019ce6.slice/crio-0fb6dc122b8a45303c5d860a79ac869751865f761ae822631afd85e4ccdf5aff WatchSource:0}: Error finding container 0fb6dc122b8a45303c5d860a79ac869751865f761ae822631afd85e4ccdf5aff: Status 404 returned error can't find the container with id 0fb6dc122b8a45303c5d860a79ac869751865f761ae822631afd85e4ccdf5aff
Sep 18 16:27:06.537490 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:06.537457    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x" event={"ID":"51b4ba4c-8f90-4ef2-996d-5883e6019ce6","Type":"ContainerStarted","Data":"6ccdbcb9459ef747c24f741a5340ee4314a2dfd5a3acd8c09783527e1fd978d2"}
Sep 18 16:27:06.537621 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:06.537509    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x" event={"ID":"51b4ba4c-8f90-4ef2-996d-5883e6019ce6","Type":"ContainerStarted","Data":"0fb6dc122b8a45303c5d860a79ac869751865f761ae822631afd85e4ccdf5aff"}
Sep 18 16:27:06.557966 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:06.557910    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x" podStartSLOduration=1.5578970810000001 podStartE2EDuration="1.557897081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:27:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:27:06.555968064 +0000 UTC m=+1322.464748427" watchObservedRunningTime="2026-09-18 16:27:06.557897081 +0000 UTC m=+1322.466677489"
Sep 18 16:27:07.545073 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:07.545020    2546 generic.go:356] "Generic (PLEG): container finished" podID="51b4ba4c-8f90-4ef2-996d-5883e6019ce6" containerID="6ccdbcb9459ef747c24f741a5340ee4314a2dfd5a3acd8c09783527e1fd978d2" exitCode=0
Sep 18 16:27:07.545432 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:07.545100    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x" event={"ID":"51b4ba4c-8f90-4ef2-996d-5883e6019ce6","Type":"ContainerDied","Data":"6ccdbcb9459ef747c24f741a5340ee4314a2dfd5a3acd8c09783527e1fd978d2"}
Sep 18 16:27:08.707776 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:08.707753    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x"
Sep 18 16:27:08.785313 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:08.785285    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b4ba4c-8f90-4ef2-996d-5883e6019ce6-maas-api-service-ca\") pod \"51b4ba4c-8f90-4ef2-996d-5883e6019ce6\" (UID: \"51b4ba4c-8f90-4ef2-996d-5883e6019ce6\") "
Sep 18 16:27:08.785591 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:08.785572    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b4ba4c-8f90-4ef2-996d-5883e6019ce6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51b4ba4c-8f90-4ef2-996d-5883e6019ce6" (UID: "51b4ba4c-8f90-4ef2-996d-5883e6019ce6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:27:08.886572 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:08.886500    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b4ba4c-8f90-4ef2-996d-5883e6019ce6-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:09.565610 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:09.565569    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x"
Sep 18 16:27:09.565790 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:09.565602    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x" event={"ID":"51b4ba4c-8f90-4ef2-996d-5883e6019ce6","Type":"ContainerDied","Data":"0fb6dc122b8a45303c5d860a79ac869751865f761ae822631afd85e4ccdf5aff"}
Sep 18 16:27:09.565790 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:09.565688    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fb6dc122b8a45303c5d860a79ac869751865f761ae822631afd85e4ccdf5aff"
Sep 18 16:27:10.436905 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.436848    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:27:10.503224 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.503195    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-credential-socket\") pod \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") "
Sep 18 16:27:10.503378 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.503263    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-podinfo\") pod \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") "
Sep 18 16:27:10.503378 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.503296    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-workload-certs\") pod \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") "
Sep 18 16:27:10.503502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.503377    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-token\") pod \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") "
Sep 18 16:27:10.503502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.503402    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-workload-socket\") pod \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") "
Sep 18 16:27:10.503502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.503439    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istiod-ca-cert\") pod \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") "
Sep 18 16:27:10.503502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.503466    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-data\") pod \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") "
Sep 18 16:27:10.503502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.503480    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "72164f1c-30e8-493b-9bd2-238b8cb49fa4" (UID: "72164f1c-30e8-493b-9bd2-238b8cb49fa4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:27:10.503759 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.503527    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-envoy\") pod \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") "
Sep 18 16:27:10.503759 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.503636    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhb6l\" (UniqueName: \"kubernetes.io/projected/72164f1c-30e8-493b-9bd2-238b8cb49fa4-kube-api-access-zhb6l\") pod \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\" (UID: \"72164f1c-30e8-493b-9bd2-238b8cb49fa4\") "
Sep 18 16:27:10.503759 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.503665    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "72164f1c-30e8-493b-9bd2-238b8cb49fa4" (UID: "72164f1c-30e8-493b-9bd2-238b8cb49fa4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:27:10.503921 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.503872    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-data" (OuterVolumeSpecName: "istio-data") pod "72164f1c-30e8-493b-9bd2-238b8cb49fa4" (UID: "72164f1c-30e8-493b-9bd2-238b8cb49fa4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:27:10.504037 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.503999    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "72164f1c-30e8-493b-9bd2-238b8cb49fa4" (UID: "72164f1c-30e8-493b-9bd2-238b8cb49fa4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:27:10.504218 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.504175    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "72164f1c-30e8-493b-9bd2-238b8cb49fa4" (UID: "72164f1c-30e8-493b-9bd2-238b8cb49fa4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:27:10.504556 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.504534    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-credential-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:10.504637 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.504560    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-workload-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:10.504637 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.504573    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-workload-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:10.504637 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.504585    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istiod-ca-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:10.504637 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.504597    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-data\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:10.505565 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.505540    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "72164f1c-30e8-493b-9bd2-238b8cb49fa4" (UID: "72164f1c-30e8-493b-9bd2-238b8cb49fa4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:27:10.506009 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.505976    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-token" (OuterVolumeSpecName: "istio-token") pod "72164f1c-30e8-493b-9bd2-238b8cb49fa4" (UID: "72164f1c-30e8-493b-9bd2-238b8cb49fa4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:27:10.506239 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.506216    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "72164f1c-30e8-493b-9bd2-238b8cb49fa4" (UID: "72164f1c-30e8-493b-9bd2-238b8cb49fa4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:27:10.506307 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.506245    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72164f1c-30e8-493b-9bd2-238b8cb49fa4-kube-api-access-zhb6l" (OuterVolumeSpecName: "kube-api-access-zhb6l") pod "72164f1c-30e8-493b-9bd2-238b8cb49fa4" (UID: "72164f1c-30e8-493b-9bd2-238b8cb49fa4"). InnerVolumeSpecName "kube-api-access-zhb6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:27:10.573533 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.573499    2546 generic.go:356] "Generic (PLEG): container finished" podID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerID="bb6a62dcf07783f0e3dabafabb53568d789f4ef1690887f54fb8dcd17743a2b9" exitCode=0
Sep 18 16:27:10.573658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.573552    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"
Sep 18 16:27:10.573658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.573552    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" event={"ID":"72164f1c-30e8-493b-9bd2-238b8cb49fa4","Type":"ContainerDied","Data":"bb6a62dcf07783f0e3dabafabb53568d789f4ef1690887f54fb8dcd17743a2b9"}
Sep 18 16:27:10.573658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.573626    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t" event={"ID":"72164f1c-30e8-493b-9bd2-238b8cb49fa4","Type":"ContainerDied","Data":"a0fc941e8bbce681eac809180b006c53f75d3b0f414d5d4cf08d59f3061b7ae6"}
Sep 18 16:27:10.573658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.573656    2546 scope.go:117] "RemoveContainer" containerID="bb6a62dcf07783f0e3dabafabb53568d789f4ef1690887f54fb8dcd17743a2b9"
Sep 18 16:27:10.584471 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.584447    2546 scope.go:117] "RemoveContainer" containerID="abb4387fcaff06156db15a56cb8549eef34b15b779a781289d96c04ac290a56b"
Sep 18 16:27:10.592730 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.592716    2546 scope.go:117] "RemoveContainer" containerID="bb6a62dcf07783f0e3dabafabb53568d789f4ef1690887f54fb8dcd17743a2b9"
Sep 18 16:27:10.592981 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:27:10.592963    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb6a62dcf07783f0e3dabafabb53568d789f4ef1690887f54fb8dcd17743a2b9\": container with ID starting with bb6a62dcf07783f0e3dabafabb53568d789f4ef1690887f54fb8dcd17743a2b9 not found: ID does not exist" containerID="bb6a62dcf07783f0e3dabafabb53568d789f4ef1690887f54fb8dcd17743a2b9"
Sep 18 16:27:10.593032 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.592988    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb6a62dcf07783f0e3dabafabb53568d789f4ef1690887f54fb8dcd17743a2b9"} err="failed to get container status \"bb6a62dcf07783f0e3dabafabb53568d789f4ef1690887f54fb8dcd17743a2b9\": rpc error: code = NotFound desc = could not find container \"bb6a62dcf07783f0e3dabafabb53568d789f4ef1690887f54fb8dcd17743a2b9\": container with ID starting with bb6a62dcf07783f0e3dabafabb53568d789f4ef1690887f54fb8dcd17743a2b9 not found: ID does not exist"
Sep 18 16:27:10.593032 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.593005    2546 scope.go:117] "RemoveContainer" containerID="abb4387fcaff06156db15a56cb8549eef34b15b779a781289d96c04ac290a56b"
Sep 18 16:27:10.593263 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:27:10.593246    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb4387fcaff06156db15a56cb8549eef34b15b779a781289d96c04ac290a56b\": container with ID starting with abb4387fcaff06156db15a56cb8549eef34b15b779a781289d96c04ac290a56b not found: ID does not exist" containerID="abb4387fcaff06156db15a56cb8549eef34b15b779a781289d96c04ac290a56b"
Sep 18 16:27:10.593302 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.593270    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb4387fcaff06156db15a56cb8549eef34b15b779a781289d96c04ac290a56b"} err="failed to get container status \"abb4387fcaff06156db15a56cb8549eef34b15b779a781289d96c04ac290a56b\": rpc error: code = NotFound desc = could not find container \"abb4387fcaff06156db15a56cb8549eef34b15b779a781289d96c04ac290a56b\": container with ID starting with abb4387fcaff06156db15a56cb8549eef34b15b779a781289d96c04ac290a56b not found: ID does not exist"
Sep 18 16:27:10.602895 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.602875    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"]
Sep 18 16:27:10.605406 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.605385    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-podinfo\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:10.605406 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.605405    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-token\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:10.605556 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.605414    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72164f1c-30e8-493b-9bd2-238b8cb49fa4-istio-envoy\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:10.605556 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.605423    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhb6l\" (UniqueName: \"kubernetes.io/projected/72164f1c-30e8-493b-9bd2-238b8cb49fa4-kube-api-access-zhb6l\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:10.606340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:10.606321    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fbc529-openshift-default-576fc5c9b-7lq8t"]
Sep 18 16:27:12.076961 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.076922    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"]
Sep 18 16:27:12.077474 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.077451    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b4ba4c-8f90-4ef2-996d-5883e6019ce6" containerName="revoke-keys"
Sep 18 16:27:12.077563 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.077478    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b4ba4c-8f90-4ef2-996d-5883e6019ce6" containerName="revoke-keys"
Sep 18 16:27:12.077563 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.077494    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy"
Sep 18 16:27:12.077563 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.077503    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy"
Sep 18 16:27:12.077563 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.077523    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy"
Sep 18 16:27:12.077563 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.077532    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy"
Sep 18 16:27:12.077749 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.077654    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy"
Sep 18 16:27:12.077749 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.077664    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b4ba4c-8f90-4ef2-996d-5883e6019ce6" containerName="revoke-keys"
Sep 18 16:27:12.077749 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.077670    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" containerName="istio-proxy"
Sep 18 16:27:12.082254 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.082238    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.084999 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.084978    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4f2d8a\"/\"e2e-conflict-b-5b8cb121-kserve-self-signed-certs\""
Sep 18 16:27:12.097506 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.097483    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"]
Sep 18 16:27:12.219088 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.219056    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-dshm\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.219255 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.219106    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-tmp-dir\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.219255 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.219146    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-model-cache\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.219255 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.219178    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-home\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.219433 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.219256    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhc5j\" (UniqueName: \"kubernetes.io/projected/def79537-f72d-4278-939b-b62fe507e6e1-kube-api-access-dhc5j\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.219433 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.219314    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def79537-f72d-4278-939b-b62fe507e6e1-tls-certs\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.320757 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.320726    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhc5j\" (UniqueName: \"kubernetes.io/projected/def79537-f72d-4278-939b-b62fe507e6e1-kube-api-access-dhc5j\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.320907 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.320773    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def79537-f72d-4278-939b-b62fe507e6e1-tls-certs\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.320907 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.320843    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-dshm\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.320907 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.320860    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-tmp-dir\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.321081 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.320978    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-model-cache\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.321081 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.321057    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-home\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.321273 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.321245    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-tmp-dir\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.321412 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.321305    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-model-cache\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.321412 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.321397    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-home\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.323149 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.323129    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-dshm\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.323472 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.323455    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def79537-f72d-4278-939b-b62fe507e6e1-tls-certs\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.330881 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.330826    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhc5j\" (UniqueName: \"kubernetes.io/projected/def79537-f72d-4278-939b-b62fe507e6e1-kube-api-access-dhc5j\") pod \"e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") " pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.391772 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.391742    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:12.602365 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.602287    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72164f1c-30e8-493b-9bd2-238b8cb49fa4" path="/var/lib/kubelet/pods/72164f1c-30e8-493b-9bd2-238b8cb49fa4/volumes"
Sep 18 16:27:12.668166 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:12.668146    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"]
Sep 18 16:27:12.670666 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:27:12.670635    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddef79537_f72d_4278_939b_b62fe507e6e1.slice/crio-be34391894a8851576da1e718a6beca838a2f3f2fabf3f5a1f65761f3585a6ca WatchSource:0}: Error finding container be34391894a8851576da1e718a6beca838a2f3f2fabf3f5a1f65761f3585a6ca: Status 404 returned error can't find the container with id be34391894a8851576da1e718a6beca838a2f3f2fabf3f5a1f65761f3585a6ca
Sep 18 16:27:13.595248 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:13.595204    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv" event={"ID":"def79537-f72d-4278-939b-b62fe507e6e1","Type":"ContainerStarted","Data":"b612f71bd647aef4b4bc6d7acf2ffbd3194b84901c6a0ccf32a30a30e5bafbc6"}
Sep 18 16:27:13.595615 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:13.595264    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv" event={"ID":"def79537-f72d-4278-939b-b62fe507e6e1","Type":"ContainerStarted","Data":"be34391894a8851576da1e718a6beca838a2f3f2fabf3f5a1f65761f3585a6ca"}
Sep 18 16:27:13.622161 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:13.622108    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv" podStartSLOduration=1.622090877 podStartE2EDuration="1.622090877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:27:13.621376757 +0000 UTC m=+1329.530157109" watchObservedRunningTime="2026-09-18 16:27:13.622090877 +0000 UTC m=+1329.530871229"
Sep 18 16:27:14.958042 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:14.957998    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"]
Sep 18 16:27:17.364546 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:17.364503    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x"]
Sep 18 16:27:17.372897 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:17.372870    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f22789-w6q4x"]
Sep 18 16:27:18.604070 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:18.604005    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51b4ba4c-8f90-4ef2-996d-5883e6019ce6" path="/var/lib/kubelet/pods/51b4ba4c-8f90-4ef2-996d-5883e6019ce6/volumes"
Sep 18 16:27:19.785264 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:19.785230    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:27:22.392409 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:22.392376    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:22.392874 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:22.392747    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:22.404763 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:22.404743    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:22.658476 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:22.658402    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:27:29.976569 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:29.976470    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs" podUID="6db7a1cf-4aa6-4c88-9778-9da8485536a2" containerName="main" containerID="cri-o://b864b2b111b479710c7f9bdfd4007dee38d4aa4d778a140c5034af707e5e8966" gracePeriod=45
Sep 18 16:27:30.228927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.228869    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:27:30.311628 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.311601    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24rbk\" (UniqueName: \"kubernetes.io/projected/6db7a1cf-4aa6-4c88-9778-9da8485536a2-kube-api-access-24rbk\") pod \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") "
Sep 18 16:27:30.311781 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.311669    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db7a1cf-4aa6-4c88-9778-9da8485536a2-tls-certs\") pod \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") "
Sep 18 16:27:30.311781 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.311743    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-home\") pod \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") "
Sep 18 16:27:30.311781 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.311773    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-tmp-dir\") pod \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") "
Sep 18 16:27:30.311935 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.311835    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-dshm\") pod \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") "
Sep 18 16:27:30.311935 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.311882    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-model-cache\") pod \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\" (UID: \"6db7a1cf-4aa6-4c88-9778-9da8485536a2\") "
Sep 18 16:27:30.312072 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.312007    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-home" (OuterVolumeSpecName: "home") pod "6db7a1cf-4aa6-4c88-9778-9da8485536a2" (UID: "6db7a1cf-4aa6-4c88-9778-9da8485536a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:27:30.312144 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.312087    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6db7a1cf-4aa6-4c88-9778-9da8485536a2" (UID: "6db7a1cf-4aa6-4c88-9778-9da8485536a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:27:30.312258 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.312235    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "6db7a1cf-4aa6-4c88-9778-9da8485536a2" (UID: "6db7a1cf-4aa6-4c88-9778-9da8485536a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:27:30.312422 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.312387    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:30.312422 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.312411    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:30.312422 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.312423    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:30.313868 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.313838    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db7a1cf-4aa6-4c88-9778-9da8485536a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6db7a1cf-4aa6-4c88-9778-9da8485536a2" (UID: "6db7a1cf-4aa6-4c88-9778-9da8485536a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:27:30.313943 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.313871    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db7a1cf-4aa6-4c88-9778-9da8485536a2-kube-api-access-24rbk" (OuterVolumeSpecName: "kube-api-access-24rbk") pod "6db7a1cf-4aa6-4c88-9778-9da8485536a2" (UID: "6db7a1cf-4aa6-4c88-9778-9da8485536a2"). InnerVolumeSpecName "kube-api-access-24rbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:27:30.313943 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.313888    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-dshm" (OuterVolumeSpecName: "dshm") pod "6db7a1cf-4aa6-4c88-9778-9da8485536a2" (UID: "6db7a1cf-4aa6-4c88-9778-9da8485536a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:27:30.413392 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.413366    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6db7a1cf-4aa6-4c88-9778-9da8485536a2-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:30.413392 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.413388    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6db7a1cf-4aa6-4c88-9778-9da8485536a2-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:30.413537 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.413399    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24rbk\" (UniqueName: \"kubernetes.io/projected/6db7a1cf-4aa6-4c88-9778-9da8485536a2-kube-api-access-24rbk\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:27:30.692744 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.692711    2546 generic.go:356] "Generic (PLEG): container finished" podID="6db7a1cf-4aa6-4c88-9778-9da8485536a2" containerID="b864b2b111b479710c7f9bdfd4007dee38d4aa4d778a140c5034af707e5e8966" exitCode=0
Sep 18 16:27:30.692908 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.692754    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs" event={"ID":"6db7a1cf-4aa6-4c88-9778-9da8485536a2","Type":"ContainerDied","Data":"b864b2b111b479710c7f9bdfd4007dee38d4aa4d778a140c5034af707e5e8966"}
Sep 18 16:27:30.692908 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.692759    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"
Sep 18 16:27:30.692908 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.692781    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs" event={"ID":"6db7a1cf-4aa6-4c88-9778-9da8485536a2","Type":"ContainerDied","Data":"177c8f6b804a8265ea3d1eecd1af0b7b40074812ea66a187c47508160dcc7f5c"}
Sep 18 16:27:30.692908 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.692797    2546 scope.go:117] "RemoveContainer" containerID="b864b2b111b479710c7f9bdfd4007dee38d4aa4d778a140c5034af707e5e8966"
Sep 18 16:27:30.701819 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.701800    2546 scope.go:117] "RemoveContainer" containerID="b864b2b111b479710c7f9bdfd4007dee38d4aa4d778a140c5034af707e5e8966"
Sep 18 16:27:30.702126 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:27:30.702106    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b864b2b111b479710c7f9bdfd4007dee38d4aa4d778a140c5034af707e5e8966\": container with ID starting with b864b2b111b479710c7f9bdfd4007dee38d4aa4d778a140c5034af707e5e8966 not found: ID does not exist" containerID="b864b2b111b479710c7f9bdfd4007dee38d4aa4d778a140c5034af707e5e8966"
Sep 18 16:27:30.702196 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.702134    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b864b2b111b479710c7f9bdfd4007dee38d4aa4d778a140c5034af707e5e8966"} err="failed to get container status \"b864b2b111b479710c7f9bdfd4007dee38d4aa4d778a140c5034af707e5e8966\": rpc error: code = NotFound desc = could not find container \"b864b2b111b479710c7f9bdfd4007dee38d4aa4d778a140c5034af707e5e8966\": container with ID starting with b864b2b111b479710c7f9bdfd4007dee38d4aa4d778a140c5034af707e5e8966 not found: ID does not exist"
Sep 18 16:27:30.714821 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.714792    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"]
Sep 18 16:27:30.719776 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:30.719754    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f22789/test-model-f22789-kserve-7b46b4d8b9-hgbbs"]
Sep 18 16:27:32.603498 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:32.603471    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6db7a1cf-4aa6-4c88-9778-9da8485536a2" path="/var/lib/kubelet/pods/6db7a1cf-4aa6-4c88-9778-9da8485536a2/volumes"
Sep 18 16:27:43.156375 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.156344    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"]
Sep 18 16:27:43.156732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.156726    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6db7a1cf-4aa6-4c88-9778-9da8485536a2" containerName="main"
Sep 18 16:27:43.156772 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.156736    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6db7a1cf-4aa6-4c88-9778-9da8485536a2" containerName="main"
Sep 18 16:27:43.156806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.156799    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6db7a1cf-4aa6-4c88-9778-9da8485536a2" containerName="main"
Sep 18 16:27:43.159893 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.159877    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.163616 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.163590    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-157a333a\"/\"e2e-lifecycle-model-157a333a-kserve-self-signed-certs\""
Sep 18 16:27:43.164732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.164708    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-157a333a\"/\"kube-root-ca.crt\""
Sep 18 16:27:43.164865 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.164773    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-157a333a\"/\"default-dockercfg-prgz2\""
Sep 18 16:27:43.167662 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.167633    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-157a333a\"/\"openshift-service-ca.crt\""
Sep 18 16:27:43.175575 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.175553    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"]
Sep 18 16:27:43.236591 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.236561    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-home\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.236591 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.236593    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-dshm\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.236771 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.236658    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt7rq\" (UniqueName: \"kubernetes.io/projected/12f01686-2697-4d23-aed6-d4911b18ce37-kube-api-access-mt7rq\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.236771 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.236704    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-model-cache\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.236771 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.236722    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-tmp-dir\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.236771 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.236748    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f01686-2697-4d23-aed6-d4911b18ce37-tls-certs\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.337974 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.337928    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-home\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.338266 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.338239    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-dshm\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.338405 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.338377    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-home\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.338628 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.338595    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt7rq\" (UniqueName: \"kubernetes.io/projected/12f01686-2697-4d23-aed6-d4911b18ce37-kube-api-access-mt7rq\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.338780 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.338766    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-model-cache\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.338888 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.338876    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-tmp-dir\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.339007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.338996    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f01686-2697-4d23-aed6-d4911b18ce37-tls-certs\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.339374 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.339087    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-model-cache\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.339374 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.339337    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-tmp-dir\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.342162 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.342139    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-dshm\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.342987 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.342965    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f01686-2697-4d23-aed6-d4911b18ce37-tls-certs\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.354811 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.354785    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt7rq\" (UniqueName: \"kubernetes.io/projected/12f01686-2697-4d23-aed6-d4911b18ce37-kube-api-access-mt7rq\") pod \"e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") " pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.470352 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.470326    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:43.755086 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.755061    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"]
Sep 18 16:27:43.756232 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:27:43.756209    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12f01686_2697_4d23_aed6_d4911b18ce37.slice/crio-2859fe22b5aa7b9fd057173857613605bfd43ed59c56a9934e76068b08ccc3e6 WatchSource:0}: Error finding container 2859fe22b5aa7b9fd057173857613605bfd43ed59c56a9934e76068b08ccc3e6: Status 404 returned error can't find the container with id 2859fe22b5aa7b9fd057173857613605bfd43ed59c56a9934e76068b08ccc3e6
Sep 18 16:27:43.768785 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:43.768757    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6" event={"ID":"12f01686-2697-4d23-aed6-d4911b18ce37","Type":"ContainerStarted","Data":"2859fe22b5aa7b9fd057173857613605bfd43ed59c56a9934e76068b08ccc3e6"}
Sep 18 16:27:44.749649 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:44.749618    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:27:44.776481 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:44.776453    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6" event={"ID":"12f01686-2697-4d23-aed6-d4911b18ce37","Type":"ContainerStarted","Data":"9d49de3a91742d343bba2ef04bdd249c4c7ff9c3c0aee98fa5084503e9449751"}
Sep 18 16:27:44.802117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:44.802065    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6" podStartSLOduration=1.802049839 podStartE2EDuration="1.802049839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:27:44.800113219 +0000 UTC m=+1360.708893570" watchObservedRunningTime="2026-09-18 16:27:44.802049839 +0000 UTC m=+1360.710830186"
Sep 18 16:27:53.470899 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:53.470866    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:53.471290 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:53.471164    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:53.482633 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:53.482613    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:27:53.842917 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:27:53.842835    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:28:01.268312 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:01.268278    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"]
Sep 18 16:28:01.268891 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:01.268468    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy" containerID="cri-o://3412385dde8196da5af73dc41bc60a8e3a4eab080fb951d9afc6e397b341e67b" gracePeriod=30
Sep 18 16:28:02.131019 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.130982    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"]
Sep 18 16:28:02.135947 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.135927    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.138782 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.138760    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-a75117-openshift-default-dockercfg-x6fzh\""
Sep 18 16:28:02.151095 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.151074    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"]
Sep 18 16:28:02.233699 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.233104    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822e99a6-4dbb-43d9-b068-7237dd24b74c-istio-podinfo\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.233699 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.233172    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqqk\" (UniqueName: \"kubernetes.io/projected/822e99a6-4dbb-43d9-b068-7237dd24b74c-kube-api-access-2xqqk\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.233699 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.233216    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-workload-socket\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.233699 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.233239    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-credential-socket\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.233699 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.233292    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822e99a6-4dbb-43d9-b068-7237dd24b74c-istio-token\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.233699 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.233334    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-istio-data\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.233699 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.233379    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822e99a6-4dbb-43d9-b068-7237dd24b74c-istiod-ca-cert\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.233699 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.233427    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-istio-envoy\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.233699 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.233575    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-workload-certs\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.334106 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.334073    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqqk\" (UniqueName: \"kubernetes.io/projected/822e99a6-4dbb-43d9-b068-7237dd24b74c-kube-api-access-2xqqk\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.334548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.334120    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-workload-socket\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.334548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.334136    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-credential-socket\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.334548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.334173    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822e99a6-4dbb-43d9-b068-7237dd24b74c-istio-token\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.334548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.334202    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-istio-data\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.334548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.334232    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822e99a6-4dbb-43d9-b068-7237dd24b74c-istiod-ca-cert\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.334548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.334275    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-istio-envoy\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.334548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.334369    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-workload-certs\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.334548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.334423    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822e99a6-4dbb-43d9-b068-7237dd24b74c-istio-podinfo\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.334548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.334544    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-workload-socket\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.335073 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.334692    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-credential-socket\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.335073 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.334765    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-istio-data\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.335073 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.334825    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-workload-certs\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.335235 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.335124    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/822e99a6-4dbb-43d9-b068-7237dd24b74c-istiod-ca-cert\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.336548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.336529    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/822e99a6-4dbb-43d9-b068-7237dd24b74c-istio-envoy\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.336770 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.336754    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/822e99a6-4dbb-43d9-b068-7237dd24b74c-istio-podinfo\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.342725 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.342699    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqqk\" (UniqueName: \"kubernetes.io/projected/822e99a6-4dbb-43d9-b068-7237dd24b74c-kube-api-access-2xqqk\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.344102 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.344081    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/822e99a6-4dbb-43d9-b068-7237dd24b74c-istio-token\") pod \"e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8\" (UID: \"822e99a6-4dbb-43d9-b068-7237dd24b74c\") " pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.450530 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.450498    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:02.742518 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.740291    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"]
Sep 18 16:28:02.743204 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:28:02.743175    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822e99a6_4dbb_43d9_b068_7237dd24b74c.slice/crio-15d1e078c8f1c6408c6a125bb1882c12eb5860b7d08b2357ca292f7f19eef2e6 WatchSource:0}: Error finding container 15d1e078c8f1c6408c6a125bb1882c12eb5860b7d08b2357ca292f7f19eef2e6: Status 404 returned error can't find the container with id 15d1e078c8f1c6408c6a125bb1882c12eb5860b7d08b2357ca292f7f19eef2e6
Sep 18 16:28:02.745437 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.745395    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:28:02.745505 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.745485    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:28:02.745544 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.745523    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:28:02.885789 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.885752    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8" event={"ID":"822e99a6-4dbb-43d9-b068-7237dd24b74c","Type":"ContainerStarted","Data":"1ef8b554dd340eea7e577be2a53149b39c07e1e2504a66bf5dc628ce5116527f"}
Sep 18 16:28:02.885985 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.885812    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8" event={"ID":"822e99a6-4dbb-43d9-b068-7237dd24b74c","Type":"ContainerStarted","Data":"15d1e078c8f1c6408c6a125bb1882c12eb5860b7d08b2357ca292f7f19eef2e6"}
Sep 18 16:28:02.911836 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:02.911789    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8" podStartSLOduration=0.911774598 podStartE2EDuration="911.774598ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:28:02.909211152 +0000 UTC m=+1378.817991503" watchObservedRunningTime="2026-09-18 16:28:02.911774598 +0000 UTC m=+1378.820554948"
Sep 18 16:28:03.450665 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:03.450633    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:03.455705 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:03.455681    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:03.893491 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:03.893401    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:03.894222 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:03.894202    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:04.829624 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:04.829589    2546 scope.go:117] "RemoveContainer" containerID="ebfe849a043aec4aa3c61ab4e654e6eb39b0ba6190cf74331d94326c187adc23"
Sep 18 16:28:04.838650 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:04.838629    2546 scope.go:117] "RemoveContainer" containerID="02f40b0c4012900cc3ecdda64ec3731ec516933cb1e96180836fbe49f20684ed"
Sep 18 16:28:05.256052 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:05.255986    2546 patch_prober.go:28] interesting pod/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 16:28:05.256217 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:05.256079    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:28:06.526264 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.526239    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:28:06.578428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.578359    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc23679e-5471-4e1c-8bb8-714173084201-istio-podinfo\") pod \"dc23679e-5471-4e1c-8bb8-714173084201\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") "
Sep 18 16:28:06.578428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.578405    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7ldn\" (UniqueName: \"kubernetes.io/projected/dc23679e-5471-4e1c-8bb8-714173084201-kube-api-access-f7ldn\") pod \"dc23679e-5471-4e1c-8bb8-714173084201\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") "
Sep 18 16:28:06.578610 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.578487    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-credential-socket\") pod \"dc23679e-5471-4e1c-8bb8-714173084201\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") "
Sep 18 16:28:06.578610 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.578516    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc23679e-5471-4e1c-8bb8-714173084201-istiod-ca-cert\") pod \"dc23679e-5471-4e1c-8bb8-714173084201\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") "
Sep 18 16:28:06.578610 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.578560    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-workload-certs\") pod \"dc23679e-5471-4e1c-8bb8-714173084201\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") "
Sep 18 16:28:06.578610 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.578598    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-workload-socket\") pod \"dc23679e-5471-4e1c-8bb8-714173084201\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") "
Sep 18 16:28:06.578816 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.578620    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-istio-envoy\") pod \"dc23679e-5471-4e1c-8bb8-714173084201\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") "
Sep 18 16:28:06.578816 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.578705    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc23679e-5471-4e1c-8bb8-714173084201-istio-token\") pod \"dc23679e-5471-4e1c-8bb8-714173084201\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") "
Sep 18 16:28:06.578816 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.578764    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-istio-data\") pod \"dc23679e-5471-4e1c-8bb8-714173084201\" (UID: \"dc23679e-5471-4e1c-8bb8-714173084201\") "
Sep 18 16:28:06.578992 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.578965    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dc23679e-5471-4e1c-8bb8-714173084201" (UID: "dc23679e-5471-4e1c-8bb8-714173084201"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:28:06.579167 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.579083    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dc23679e-5471-4e1c-8bb8-714173084201" (UID: "dc23679e-5471-4e1c-8bb8-714173084201"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:28:06.579287 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.579178    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-istio-data" (OuterVolumeSpecName: "istio-data") pod "dc23679e-5471-4e1c-8bb8-714173084201" (UID: "dc23679e-5471-4e1c-8bb8-714173084201"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:28:06.579287 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.579199    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-workload-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:28:06.579287 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.579220    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-workload-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:28:06.579445 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.579343    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dc23679e-5471-4e1c-8bb8-714173084201" (UID: "dc23679e-5471-4e1c-8bb8-714173084201"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:28:06.579812 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.579619    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc23679e-5471-4e1c-8bb8-714173084201-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dc23679e-5471-4e1c-8bb8-714173084201" (UID: "dc23679e-5471-4e1c-8bb8-714173084201"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:28:06.581359 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.581335    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc23679e-5471-4e1c-8bb8-714173084201-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dc23679e-5471-4e1c-8bb8-714173084201" (UID: "dc23679e-5471-4e1c-8bb8-714173084201"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:28:06.581645 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.581617    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc23679e-5471-4e1c-8bb8-714173084201-istio-token" (OuterVolumeSpecName: "istio-token") pod "dc23679e-5471-4e1c-8bb8-714173084201" (UID: "dc23679e-5471-4e1c-8bb8-714173084201"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:28:06.581892 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.581875    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc23679e-5471-4e1c-8bb8-714173084201-kube-api-access-f7ldn" (OuterVolumeSpecName: "kube-api-access-f7ldn") pod "dc23679e-5471-4e1c-8bb8-714173084201" (UID: "dc23679e-5471-4e1c-8bb8-714173084201"). InnerVolumeSpecName "kube-api-access-f7ldn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:28:06.582150 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.582131    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dc23679e-5471-4e1c-8bb8-714173084201" (UID: "dc23679e-5471-4e1c-8bb8-714173084201"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:28:06.680932 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.680899    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc23679e-5471-4e1c-8bb8-714173084201-istio-token\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:28:06.680932 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.680933    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-istio-data\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:28:06.681135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.680946    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc23679e-5471-4e1c-8bb8-714173084201-istio-podinfo\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:28:06.681135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.680986    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7ldn\" (UniqueName: \"kubernetes.io/projected/dc23679e-5471-4e1c-8bb8-714173084201-kube-api-access-f7ldn\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:28:06.681135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.681002    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-credential-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:28:06.681135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.681014    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc23679e-5471-4e1c-8bb8-714173084201-istiod-ca-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:28:06.681135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.681043    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc23679e-5471-4e1c-8bb8-714173084201-istio-envoy\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:28:06.911884 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.911797    2546 generic.go:356] "Generic (PLEG): container finished" podID="dc23679e-5471-4e1c-8bb8-714173084201" containerID="3412385dde8196da5af73dc41bc60a8e3a4eab080fb951d9afc6e397b341e67b" exitCode=0
Sep 18 16:28:06.911884 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.911860    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"
Sep 18 16:28:06.912163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.911889    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" event={"ID":"dc23679e-5471-4e1c-8bb8-714173084201","Type":"ContainerDied","Data":"3412385dde8196da5af73dc41bc60a8e3a4eab080fb951d9afc6e397b341e67b"}
Sep 18 16:28:06.912163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.911983    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd" event={"ID":"dc23679e-5471-4e1c-8bb8-714173084201","Type":"ContainerDied","Data":"6779d21a009a4cab1181a36853372ea9c331dedb9336bc81602b05da4078e1a0"}
Sep 18 16:28:06.912163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.912014    2546 scope.go:117] "RemoveContainer" containerID="3412385dde8196da5af73dc41bc60a8e3a4eab080fb951d9afc6e397b341e67b"
Sep 18 16:28:06.921182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.921168    2546 scope.go:117] "RemoveContainer" containerID="27bffc9521e0bb03fb980379de41e5e56cb9b94ad8c77906dceac56fc32896f4"
Sep 18 16:28:06.929180 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.929165    2546 scope.go:117] "RemoveContainer" containerID="3412385dde8196da5af73dc41bc60a8e3a4eab080fb951d9afc6e397b341e67b"
Sep 18 16:28:06.929434 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:28:06.929411    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3412385dde8196da5af73dc41bc60a8e3a4eab080fb951d9afc6e397b341e67b\": container with ID starting with 3412385dde8196da5af73dc41bc60a8e3a4eab080fb951d9afc6e397b341e67b not found: ID does not exist" containerID="3412385dde8196da5af73dc41bc60a8e3a4eab080fb951d9afc6e397b341e67b"
Sep 18 16:28:06.929525 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.929446    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3412385dde8196da5af73dc41bc60a8e3a4eab080fb951d9afc6e397b341e67b"} err="failed to get container status \"3412385dde8196da5af73dc41bc60a8e3a4eab080fb951d9afc6e397b341e67b\": rpc error: code = NotFound desc = could not find container \"3412385dde8196da5af73dc41bc60a8e3a4eab080fb951d9afc6e397b341e67b\": container with ID starting with 3412385dde8196da5af73dc41bc60a8e3a4eab080fb951d9afc6e397b341e67b not found: ID does not exist"
Sep 18 16:28:06.929525 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.929466    2546 scope.go:117] "RemoveContainer" containerID="27bffc9521e0bb03fb980379de41e5e56cb9b94ad8c77906dceac56fc32896f4"
Sep 18 16:28:06.929675 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:28:06.929660    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27bffc9521e0bb03fb980379de41e5e56cb9b94ad8c77906dceac56fc32896f4\": container with ID starting with 27bffc9521e0bb03fb980379de41e5e56cb9b94ad8c77906dceac56fc32896f4 not found: ID does not exist" containerID="27bffc9521e0bb03fb980379de41e5e56cb9b94ad8c77906dceac56fc32896f4"
Sep 18 16:28:06.929711 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.929679    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27bffc9521e0bb03fb980379de41e5e56cb9b94ad8c77906dceac56fc32896f4"} err="failed to get container status \"27bffc9521e0bb03fb980379de41e5e56cb9b94ad8c77906dceac56fc32896f4\": rpc error: code = NotFound desc = could not find container \"27bffc9521e0bb03fb980379de41e5e56cb9b94ad8c77906dceac56fc32896f4\": container with ID starting with 27bffc9521e0bb03fb980379de41e5e56cb9b94ad8c77906dceac56fc32896f4 not found: ID does not exist"
Sep 18 16:28:06.938370 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.938351    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"]
Sep 18 16:28:06.942642 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:06.942624    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f22789-openshift-default-9c484d56d-qvnqd"]
Sep 18 16:28:08.300298 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:08.300265    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:28:08.604264 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:08.604178    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc23679e-5471-4e1c-8bb8-714173084201" path="/var/lib/kubelet/pods/dc23679e-5471-4e1c-8bb8-714173084201/volumes"
Sep 18 16:28:30.870865 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:30.870833    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"]
Sep 18 16:28:30.871303 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:30.871244    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy"
Sep 18 16:28:30.871303 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:30.871265    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy"
Sep 18 16:28:30.871376 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:30.871353    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy"
Sep 18 16:28:30.871376 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:30.871363    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy"
Sep 18 16:28:30.876067 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:30.876048    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:30.880187 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:30.879972    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1eec1d\"/\"distinct-2-w0-1eec1d-kserve-self-signed-certs\""
Sep 18 16:28:30.887432 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:30.886916    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"]
Sep 18 16:28:31.043491 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.043461    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-home\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.043658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.043507    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2691e4c-7d42-4c19-8e31-b32677395c3b-tls-certs\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.043658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.043569    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-model-cache\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.043658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.043630    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-dshm\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.043762 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.043696    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-tmp-dir\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.043887 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.043864    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbktw\" (UniqueName: \"kubernetes.io/projected/f2691e4c-7d42-4c19-8e31-b32677395c3b-kube-api-access-fbktw\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.145408 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.145324    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-tmp-dir\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.145550 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.145423    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbktw\" (UniqueName: \"kubernetes.io/projected/f2691e4c-7d42-4c19-8e31-b32677395c3b-kube-api-access-fbktw\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.145550 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.145463    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-home\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.145550 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.145481    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2691e4c-7d42-4c19-8e31-b32677395c3b-tls-certs\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.145550 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.145504    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-model-cache\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.145550 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.145543    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-dshm\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.145712 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.145670    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-tmp-dir\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.145944 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.145917    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-model-cache\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.146105 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.146091    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-home\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.147701 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.147686    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-dshm\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.147952 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.147934    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2691e4c-7d42-4c19-8e31-b32677395c3b-tls-certs\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.157336 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.157318    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbktw\" (UniqueName: \"kubernetes.io/projected/f2691e4c-7d42-4c19-8e31-b32677395c3b-kube-api-access-fbktw\") pod \"distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") " pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.192342 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.192147    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:31.473436 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:31.473409    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"]
Sep 18 16:28:31.474603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:28:31.474578    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2691e4c_7d42_4c19_8e31_b32677395c3b.slice/crio-74fde4a096988c46f8037bd4fbaf56430b5e7dacf758877e726ba4ba232a25ed WatchSource:0}: Error finding container 74fde4a096988c46f8037bd4fbaf56430b5e7dacf758877e726ba4ba232a25ed: Status 404 returned error can't find the container with id 74fde4a096988c46f8037bd4fbaf56430b5e7dacf758877e726ba4ba232a25ed
Sep 18 16:28:32.055710 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:32.055674    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt" event={"ID":"f2691e4c-7d42-4c19-8e31-b32677395c3b","Type":"ContainerStarted","Data":"e4e26a19242e67118f4fd252dc9379e33eb12347c454a6821ba929d9851cdfd3"}
Sep 18 16:28:32.056225 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:32.055722    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt" event={"ID":"f2691e4c-7d42-4c19-8e31-b32677395c3b","Type":"ContainerStarted","Data":"74fde4a096988c46f8037bd4fbaf56430b5e7dacf758877e726ba4ba232a25ed"}
Sep 18 16:28:32.082905 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:32.082852    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt" podStartSLOduration=2.082836509 podStartE2EDuration="2.082836509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:28:32.07961111 +0000 UTC m=+1407.988391462" watchObservedRunningTime="2026-09-18 16:28:32.082836509 +0000 UTC m=+1407.991616858"
Sep 18 16:28:34.405643 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:34.405491    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": read tcp 10.133.0.2:40622->10.133.0.80:15021: read: connection reset by peer" start-of-body=
Sep 18 16:28:34.405643 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:34.405554    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8" podUID="822e99a6-4dbb-43d9-b068-7237dd24b74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": read tcp 10.133.0.2:40622->10.133.0.80:15021: read: connection reset by peer"
Sep 18 16:28:35.073006 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.072974    2546 generic.go:356] "Generic (PLEG): container finished" podID="822e99a6-4dbb-43d9-b068-7237dd24b74c" containerID="1ef8b554dd340eea7e577be2a53149b39c07e1e2504a66bf5dc628ce5116527f" exitCode=0
Sep 18 16:28:35.073225 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.073047    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8" event={"ID":"822e99a6-4dbb-43d9-b068-7237dd24b74c","Type":"ContainerDied","Data":"1ef8b554dd340eea7e577be2a53149b39c07e1e2504a66bf5dc628ce5116527f"}
Sep 18 16:28:35.073484 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.073469    2546 scope.go:117] "RemoveContainer" containerID="1ef8b554dd340eea7e577be2a53149b39c07e1e2504a66bf5dc628ce5116527f"
Sep 18 16:28:35.074610 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.074576    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:28:35.074709 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.074650    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:28:35.074709 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.074681    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:28:35.645046 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.643841    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"]
Sep 18 16:28:35.650464 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.650433    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy"
Sep 18 16:28:35.650464 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.650462    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc23679e-5471-4e1c-8bb8-714173084201" containerName="istio-proxy"
Sep 18 16:28:35.652940 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.652911    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.656078 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.656057    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-11725e-openshift-default-dockercfg-btfmq\""
Sep 18 16:28:35.671790 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.671759    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"]
Sep 18 16:28:35.695046 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.693608    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-credential-socket\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.695046 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.693675    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istiod-ca-cert\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.695046 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.693731    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-workload-certs\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.695046 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.693764    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-workload-socket\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.695046 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.693794    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istio-token\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.695046 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.693951    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istio-data\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.695046 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.693997    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkjhg\" (UniqueName: \"kubernetes.io/projected/1b4328b9-39c4-4fb3-9527-c4aa699774fc-kube-api-access-fkjhg\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.695046 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.694045    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istio-envoy\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.695046 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.694112    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istio-podinfo\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.794986 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.794938    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istio-podinfo\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.795211 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.795010    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-credential-socket\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.795211 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.795074    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istiod-ca-cert\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.795211 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.795122    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-workload-certs\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.795211 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.795160    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-workload-socket\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.795211 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.795188    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istio-token\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.795476 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.795300    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istio-data\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.795476 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.795334    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjhg\" (UniqueName: \"kubernetes.io/projected/1b4328b9-39c4-4fb3-9527-c4aa699774fc-kube-api-access-fkjhg\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.795476 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.795360    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istio-envoy\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.799044 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.796214    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-workload-certs\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.799044 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.796894    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istiod-ca-cert\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.799044 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.797000    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-workload-socket\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.799044 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.797156    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istio-data\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.799044 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.797331    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-credential-socket\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.803040 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.799619    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istio-envoy\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.807040 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.803791    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istio-podinfo\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.813522 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.813492    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b4328b9-39c4-4fb3-9527-c4aa699774fc-istio-token\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.814495 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.814470    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjhg\" (UniqueName: \"kubernetes.io/projected/1b4328b9-39c4-4fb3-9527-c4aa699774fc-kube-api-access-fkjhg\") pod \"e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76\" (UID: \"1b4328b9-39c4-4fb3-9527-c4aa699774fc\") " pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:35.963880 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:35.963848    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:36.102048 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:36.101372    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8" event={"ID":"822e99a6-4dbb-43d9-b068-7237dd24b74c","Type":"ContainerStarted","Data":"49c5cb8f13f65300bc408dde9a36d946e2215089fb6a1b34f7fd44fb6c422369"}
Sep 18 16:28:36.272225 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:36.272201    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"]
Sep 18 16:28:36.273615 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:28:36.273593    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4328b9_39c4_4fb3_9527_c4aa699774fc.slice/crio-6beef5e4f96dc2e3abf155603ce228b11f92d567da5625fd1f5d3f805804b1d0 WatchSource:0}: Error finding container 6beef5e4f96dc2e3abf155603ce228b11f92d567da5625fd1f5d3f805804b1d0: Status 404 returned error can't find the container with id 6beef5e4f96dc2e3abf155603ce228b11f92d567da5625fd1f5d3f805804b1d0
Sep 18 16:28:36.276078 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:36.276045    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:28:36.276179 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:36.276111    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:28:36.276179 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:36.276139    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:28:36.451575 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:36.451541    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:36.452038 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:36.451891    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 18 16:28:36.452038 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:36.451950    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8" podUID="822e99a6-4dbb-43d9-b068-7237dd24b74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 18 16:28:37.112939 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:37.112572    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76" event={"ID":"1b4328b9-39c4-4fb3-9527-c4aa699774fc","Type":"ContainerStarted","Data":"7d21027a1dcbfd2ec31c3a8622759b4d880869b36670788c125a64235b8c91ad"}
Sep 18 16:28:37.112939 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:37.112640    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76" event={"ID":"1b4328b9-39c4-4fb3-9527-c4aa699774fc","Type":"ContainerStarted","Data":"6beef5e4f96dc2e3abf155603ce228b11f92d567da5625fd1f5d3f805804b1d0"}
Sep 18 16:28:37.140357 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:37.140311    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76" podStartSLOduration=2.140296509 podStartE2EDuration="2.140296509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:28:37.136589568 +0000 UTC m=+1413.045369919" watchObservedRunningTime="2026-09-18 16:28:37.140296509 +0000 UTC m=+1413.049076860"
Sep 18 16:28:37.451132 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:37.451104    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 18 16:28:37.451289 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:37.451160    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8" podUID="822e99a6-4dbb-43d9-b068-7237dd24b74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 18 16:28:37.964436 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:37.964403    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:37.969540 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:37.969517    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:38.119548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:38.119518    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:38.120435 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:38.120413    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:28:38.451843 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:38.451814    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 18 16:28:38.452010 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:38.451868    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8" podUID="822e99a6-4dbb-43d9-b068-7237dd24b74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 18 16:28:39.451589 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:39.451553    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 18 16:28:39.451971 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:39.451613    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8" podUID="822e99a6-4dbb-43d9-b068-7237dd24b74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 18 16:28:40.451218 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:40.451184    2546 patch_prober.go:28] interesting pod/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Sep 18 16:28:40.451391 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:40.451236    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8" podUID="822e99a6-4dbb-43d9-b068-7237dd24b74c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Sep 18 16:28:41.193334 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:41.193303    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:41.193662 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:41.193353    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:41.205135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:41.205112    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:41.454729 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:41.454696    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:41.455062 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:41.455041    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:41.455649 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:41.455627    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8"
Sep 18 16:28:41.865499 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:41.865426    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:28:42.153199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:42.153119    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:28:47.310515 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:28:47.310482    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"]
Sep 18 16:29:02.329187 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.329109    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv" podUID="def79537-f72d-4278-939b-b62fe507e6e1" containerName="main" containerID="cri-o://b612f71bd647aef4b4bc6d7acf2ffbd3194b84901c6a0ccf32a30a30e5bafbc6" gracePeriod=45
Sep 18 16:29:02.591275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.591219    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:29:02.677336 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.677309    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-model-cache\") pod \"def79537-f72d-4278-939b-b62fe507e6e1\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") "
Sep 18 16:29:02.677508 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.677383    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-tmp-dir\") pod \"def79537-f72d-4278-939b-b62fe507e6e1\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") "
Sep 18 16:29:02.677508 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.677437    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhc5j\" (UniqueName: \"kubernetes.io/projected/def79537-f72d-4278-939b-b62fe507e6e1-kube-api-access-dhc5j\") pod \"def79537-f72d-4278-939b-b62fe507e6e1\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") "
Sep 18 16:29:02.677508 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.677498    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-dshm\") pod \"def79537-f72d-4278-939b-b62fe507e6e1\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") "
Sep 18 16:29:02.677678 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.677532    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-home\") pod \"def79537-f72d-4278-939b-b62fe507e6e1\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") "
Sep 18 16:29:02.677678 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.677565    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def79537-f72d-4278-939b-b62fe507e6e1-tls-certs\") pod \"def79537-f72d-4278-939b-b62fe507e6e1\" (UID: \"def79537-f72d-4278-939b-b62fe507e6e1\") "
Sep 18 16:29:02.677678 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.677617    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "def79537-f72d-4278-939b-b62fe507e6e1" (UID: "def79537-f72d-4278-939b-b62fe507e6e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:29:02.677678 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.677637    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "def79537-f72d-4278-939b-b62fe507e6e1" (UID: "def79537-f72d-4278-939b-b62fe507e6e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:29:02.678000 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.677973    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-home" (OuterVolumeSpecName: "home") pod "def79537-f72d-4278-939b-b62fe507e6e1" (UID: "def79537-f72d-4278-939b-b62fe507e6e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:29:02.678083 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.678072    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:29:02.678140 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.678090    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:29:02.678140 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.678103    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:29:02.680337 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.680306    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-dshm" (OuterVolumeSpecName: "dshm") pod "def79537-f72d-4278-939b-b62fe507e6e1" (UID: "def79537-f72d-4278-939b-b62fe507e6e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:29:02.680466 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.680446    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def79537-f72d-4278-939b-b62fe507e6e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "def79537-f72d-4278-939b-b62fe507e6e1" (UID: "def79537-f72d-4278-939b-b62fe507e6e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:29:02.680518 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.680480    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def79537-f72d-4278-939b-b62fe507e6e1-kube-api-access-dhc5j" (OuterVolumeSpecName: "kube-api-access-dhc5j") pod "def79537-f72d-4278-939b-b62fe507e6e1" (UID: "def79537-f72d-4278-939b-b62fe507e6e1"). InnerVolumeSpecName "kube-api-access-dhc5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:29:02.778768 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.778744    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhc5j\" (UniqueName: \"kubernetes.io/projected/def79537-f72d-4278-939b-b62fe507e6e1-kube-api-access-dhc5j\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:29:02.778768 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.778766    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def79537-f72d-4278-939b-b62fe507e6e1-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:29:02.778907 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:02.778775    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def79537-f72d-4278-939b-b62fe507e6e1-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:29:03.263318 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:03.263285    2546 generic.go:356] "Generic (PLEG): container finished" podID="def79537-f72d-4278-939b-b62fe507e6e1" containerID="b612f71bd647aef4b4bc6d7acf2ffbd3194b84901c6a0ccf32a30a30e5bafbc6" exitCode=0
Sep 18 16:29:03.263488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:03.263333    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"
Sep 18 16:29:03.263488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:03.263369    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv" event={"ID":"def79537-f72d-4278-939b-b62fe507e6e1","Type":"ContainerDied","Data":"b612f71bd647aef4b4bc6d7acf2ffbd3194b84901c6a0ccf32a30a30e5bafbc6"}
Sep 18 16:29:03.263488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:03.263407    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv" event={"ID":"def79537-f72d-4278-939b-b62fe507e6e1","Type":"ContainerDied","Data":"be34391894a8851576da1e718a6beca838a2f3f2fabf3f5a1f65761f3585a6ca"}
Sep 18 16:29:03.263488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:03.263426    2546 scope.go:117] "RemoveContainer" containerID="b612f71bd647aef4b4bc6d7acf2ffbd3194b84901c6a0ccf32a30a30e5bafbc6"
Sep 18 16:29:03.274661 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:03.274642    2546 scope.go:117] "RemoveContainer" containerID="b612f71bd647aef4b4bc6d7acf2ffbd3194b84901c6a0ccf32a30a30e5bafbc6"
Sep 18 16:29:03.275202 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:29:03.275175    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b612f71bd647aef4b4bc6d7acf2ffbd3194b84901c6a0ccf32a30a30e5bafbc6\": container with ID starting with b612f71bd647aef4b4bc6d7acf2ffbd3194b84901c6a0ccf32a30a30e5bafbc6 not found: ID does not exist" containerID="b612f71bd647aef4b4bc6d7acf2ffbd3194b84901c6a0ccf32a30a30e5bafbc6"
Sep 18 16:29:03.275290 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:03.275211    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b612f71bd647aef4b4bc6d7acf2ffbd3194b84901c6a0ccf32a30a30e5bafbc6"} err="failed to get container status \"b612f71bd647aef4b4bc6d7acf2ffbd3194b84901c6a0ccf32a30a30e5bafbc6\": rpc error: code = NotFound desc = could not find container \"b612f71bd647aef4b4bc6d7acf2ffbd3194b84901c6a0ccf32a30a30e5bafbc6\": container with ID starting with b612f71bd647aef4b4bc6d7acf2ffbd3194b84901c6a0ccf32a30a30e5bafbc6 not found: ID does not exist"
Sep 18 16:29:03.293567 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:03.293524    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"]
Sep 18 16:29:03.302216 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:03.302178    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/e2e-conflict-b-5b8cb121-kserve-788cb86c6f-xz7jv"]
Sep 18 16:29:04.609833 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:04.609803    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="def79537-f72d-4278-939b-b62fe507e6e1" path="/var/lib/kubelet/pods/def79537-f72d-4278-939b-b62fe507e6e1/volumes"
Sep 18 16:29:04.870111 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:04.870020    2546 scope.go:117] "RemoveContainer" containerID="6a4655cdfe131ff9657eff7d31f3b57f918c268eb3fe094530bb0a00a874dc11"
Sep 18 16:29:07.682653 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:07.682616    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f"]
Sep 18 16:29:07.684329 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:07.684305    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def79537-f72d-4278-939b-b62fe507e6e1" containerName="main"
Sep 18 16:29:07.684491 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:07.684481    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="def79537-f72d-4278-939b-b62fe507e6e1" containerName="main"
Sep 18 16:29:07.684699 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:07.684687    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="def79537-f72d-4278-939b-b62fe507e6e1" containerName="main"
Sep 18 16:29:07.691560 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:07.691536    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f"
Sep 18 16:29:07.700428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:07.700396    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:29:07.701078 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:07.701059    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:29:07.706613 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:07.706587    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f"]
Sep 18 16:29:07.837929 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:07.837893    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a2e21a-556b-495a-bf14-67bd9aa8e1fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-157a333a-qcx4f\" (UID: \"13a2e21a-556b-495a-bf14-67bd9aa8e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f"
Sep 18 16:29:07.939556 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:07.939470    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a2e21a-556b-495a-bf14-67bd9aa8e1fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-157a333a-qcx4f\" (UID: \"13a2e21a-556b-495a-bf14-67bd9aa8e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f"
Sep 18 16:29:07.940167 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:07.940144    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a2e21a-556b-495a-bf14-67bd9aa8e1fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-157a333a-qcx4f\" (UID: \"13a2e21a-556b-495a-bf14-67bd9aa8e1fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f"
Sep 18 16:29:08.010322 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:08.010290    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f"
Sep 18 16:29:08.292102 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:08.292067    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f"]
Sep 18 16:29:08.301531 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:29:08.301494    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a2e21a_556b_495a_bf14_67bd9aa8e1fd.slice/crio-a1128c68a147dba71ed3fbffbc0a04e29c23ffd325625a3d228127c58a21e0d5 WatchSource:0}: Error finding container a1128c68a147dba71ed3fbffbc0a04e29c23ffd325625a3d228127c58a21e0d5: Status 404 returned error can't find the container with id a1128c68a147dba71ed3fbffbc0a04e29c23ffd325625a3d228127c58a21e0d5
Sep 18 16:29:08.316410 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:08.316376    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f" event={"ID":"13a2e21a-556b-495a-bf14-67bd9aa8e1fd","Type":"ContainerStarted","Data":"a1128c68a147dba71ed3fbffbc0a04e29c23ffd325625a3d228127c58a21e0d5"}
Sep 18 16:29:09.325463 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:09.325427    2546 generic.go:356] "Generic (PLEG): container finished" podID="13a2e21a-556b-495a-bf14-67bd9aa8e1fd" containerID="2f8ded875f281c5164260c7f5cbcab328c1c2e91bb6f667443200ae0d554fe42" exitCode=0
Sep 18 16:29:09.325852 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:09.325511    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f" event={"ID":"13a2e21a-556b-495a-bf14-67bd9aa8e1fd","Type":"ContainerDied","Data":"2f8ded875f281c5164260c7f5cbcab328c1c2e91bb6f667443200ae0d554fe42"}
Sep 18 16:29:10.483791 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:10.483766    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f"
Sep 18 16:29:10.577661 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:10.577629    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a2e21a-556b-495a-bf14-67bd9aa8e1fd-maas-api-service-ca\") pod \"13a2e21a-556b-495a-bf14-67bd9aa8e1fd\" (UID: \"13a2e21a-556b-495a-bf14-67bd9aa8e1fd\") "
Sep 18 16:29:10.578092 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:10.578063    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a2e21a-556b-495a-bf14-67bd9aa8e1fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13a2e21a-556b-495a-bf14-67bd9aa8e1fd" (UID: "13a2e21a-556b-495a-bf14-67bd9aa8e1fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:29:10.679626 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:10.679591    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13a2e21a-556b-495a-bf14-67bd9aa8e1fd-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:29:11.338956 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:11.338924    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f" event={"ID":"13a2e21a-556b-495a-bf14-67bd9aa8e1fd","Type":"ContainerDied","Data":"a1128c68a147dba71ed3fbffbc0a04e29c23ffd325625a3d228127c58a21e0d5"}
Sep 18 16:29:11.339131 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:11.338974    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1128c68a147dba71ed3fbffbc0a04e29c23ffd325625a3d228127c58a21e0d5"
Sep 18 16:29:11.339131 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:11.338940    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f"
Sep 18 16:29:12.346710 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:12.346676    2546 generic.go:356] "Generic (PLEG): container finished" podID="1b4328b9-39c4-4fb3-9527-c4aa699774fc" containerID="7d21027a1dcbfd2ec31c3a8622759b4d880869b36670788c125a64235b8c91ad" exitCode=0
Sep 18 16:29:12.347152 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:12.346749    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76" event={"ID":"1b4328b9-39c4-4fb3-9527-c4aa699774fc","Type":"ContainerDied","Data":"7d21027a1dcbfd2ec31c3a8622759b4d880869b36670788c125a64235b8c91ad"}
Sep 18 16:29:12.347152 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:12.347144    2546 scope.go:117] "RemoveContainer" containerID="7d21027a1dcbfd2ec31c3a8622759b4d880869b36670788c125a64235b8c91ad"
Sep 18 16:29:12.348425 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:12.348399    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:29:12.348496 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:12.348470    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:29:12.348536 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:12.348500    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:29:13.354437 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:13.354401    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76" event={"ID":"1b4328b9-39c4-4fb3-9527-c4aa699774fc","Type":"ContainerStarted","Data":"58ba88e275d17cd20f2223d42e7ec8cd5ded8a52805d3fa1d4d805a184053376"}
Sep 18 16:29:13.964910 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:13.964876    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:29:13.965114 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:13.965067    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 18 16:29:13.965171 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:13.965110    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76" podUID="1b4328b9-39c4-4fb3-9527-c4aa699774fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 18 16:29:14.964996 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:14.964962    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 18 16:29:14.965382 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:14.965016    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76" podUID="1b4328b9-39c4-4fb3-9527-c4aa699774fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 18 16:29:15.964309 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:15.964270    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 18 16:29:15.964456 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:15.964323    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76" podUID="1b4328b9-39c4-4fb3-9527-c4aa699774fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 18 16:29:16.489212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:16.489172    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:29:16.964649 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:16.964617    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 18 16:29:16.964808 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:16.964669    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76" podUID="1b4328b9-39c4-4fb3-9527-c4aa699774fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 18 16:29:17.964716 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:17.964682    2546 patch_prober.go:28] interesting pod/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Sep 18 16:29:17.965103 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:17.964739    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76" podUID="1b4328b9-39c4-4fb3-9527-c4aa699774fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Sep 18 16:29:18.968854 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:18.968824    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:29:18.969299 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:18.969059    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:29:18.969825 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:18.969809    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76"
Sep 18 16:29:25.755049 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.754997    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"]
Sep 18 16:29:25.755502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.755398    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a2e21a-556b-495a-bf14-67bd9aa8e1fd" containerName="revoke-keys"
Sep 18 16:29:25.755502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.755411    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a2e21a-556b-495a-bf14-67bd9aa8e1fd" containerName="revoke-keys"
Sep 18 16:29:25.755502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.755477    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a2e21a-556b-495a-bf14-67bd9aa8e1fd" containerName="revoke-keys"
Sep 18 16:29:25.757625 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.757605    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.767976 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.767957    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4f2d8a\"/\"distinct-w4-4f2d8a-kserve-self-signed-certs\""
Sep 18 16:29:25.842797 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.842772    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6x4\" (UniqueName: \"kubernetes.io/projected/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-kube-api-access-9f6x4\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.842909 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.842816    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-model-cache\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.842909 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.842839    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-tls-certs\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.842909 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.842867    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-home\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.843013 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.842926    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-tmp-dir\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.843068 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.843050    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-dshm\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.944280 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.944246    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-tmp-dir\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.944468 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.944401    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-dshm\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.944468 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.944447    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6x4\" (UniqueName: \"kubernetes.io/projected/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-kube-api-access-9f6x4\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.944593 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.944484    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-model-cache\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.944593 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.944507    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-tls-certs\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.944593 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.944543    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-home\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.944749 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.944631    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-tmp-dir\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.944817 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.944791    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-model-cache\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.944973 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.944955    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-home\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.946649 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.946629    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-dshm\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:25.946898 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:25.946880    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-tls-certs\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:27.049801 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:27.049767    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6x4\" (UniqueName: \"kubernetes.io/projected/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-kube-api-access-9f6x4\") pod \"distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:27.050984 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:27.050961    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"]
Sep 18 16:29:27.267608 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:27.267564    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:29.275762 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:29.275728    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"]
Sep 18 16:29:29.448782 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:29.448752    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l" event={"ID":"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a","Type":"ContainerStarted","Data":"3ccdf82b844dd9e84eb61620fe618f81dea86a5ec38d1f935724ede3bb7baadd"}
Sep 18 16:29:29.448951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:29.448792    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l" event={"ID":"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a","Type":"ContainerStarted","Data":"963c9c5ac88560b8f46c9088a3b727439e92f49ff518a8eb8b93e4579b73918c"}
Sep 18 16:29:37.268690 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:37.268659    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:37.269134 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:37.268898    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:37.280428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:37.280406    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:37.359264 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:37.359210    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l" podStartSLOduration=15.359194243 podStartE2EDuration="15.359194243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:29:30.831746296 +0000 UTC m=+1466.740526647" watchObservedRunningTime="2026-09-18 16:29:37.359194243 +0000 UTC m=+1473.267974640"
Sep 18 16:29:37.389861 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:37.389828    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f"]
Sep 18 16:29:37.391936 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:37.391917    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-157a333a-qcx4f"]
Sep 18 16:29:37.513335 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:37.513302    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:29:38.603237 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:38.603210    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13a2e21a-556b-495a-bf14-67bd9aa8e1fd" path="/var/lib/kubelet/pods/13a2e21a-556b-495a-bf14-67bd9aa8e1fd/volumes"
Sep 18 16:29:45.701096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:29:45.701060    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"]
Sep 18 16:30:00.171554 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.171520    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5865ea-29829150-6c4cr"]
Sep 18 16:30:00.174066 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.174045    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5865ea-29829150-6c4cr"
Sep 18 16:30:00.182583 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.182562    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:30:00.185148 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.185131    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5865ea-29829150-6c4cr"
Sep 18 16:30:00.190810 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.190782    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829150-djk5t"]
Sep 18 16:30:00.193988 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.193963    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5865ea-29829150-6c4cr"]
Sep 18 16:30:00.194131 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.194010    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829150-djk5t"
Sep 18 16:30:00.201696 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.201670    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf"]
Sep 18 16:30:00.204180 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.204163    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf"
Sep 18 16:30:00.213213 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.213192    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829150-jsbcr"]
Sep 18 16:30:00.216840 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.216819    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829150-jsbcr"
Sep 18 16:30:00.232508 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.232478    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829150-djk5t"]
Sep 18 16:30:00.234239 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.234218    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829150-djk5t"
Sep 18 16:30:00.236514 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.236493    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf"]
Sep 18 16:30:00.239125 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.239104    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829150-jsbcr"]
Sep 18 16:30:00.246855 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.246834    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf"
Sep 18 16:30:00.299927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.299895    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829150-jsbcr"
Sep 18 16:30:00.484890 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.484864    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5865ea-29829150-6c4cr"]
Sep 18 16:30:00.487344 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:30:00.487314    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb097701_3bef_4ba8_b4a3_fd5fda8c3dc9.slice/crio-c938ca28696e8d80a384423f98a1a6b2a9c3112c8e5d3b67a4769f932277e4eb WatchSource:0}: Error finding container c938ca28696e8d80a384423f98a1a6b2a9c3112c8e5d3b67a4769f932277e4eb: Status 404 returned error can't find the container with id c938ca28696e8d80a384423f98a1a6b2a9c3112c8e5d3b67a4769f932277e4eb
Sep 18 16:30:00.559378 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.559289    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829150-djk5t"]
Sep 18 16:30:00.562281 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:30:00.562250    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13899d7_4176_4980_88d9_93e3641f2950.slice/crio-cb2e07972eb0eb7a84d75bc017091f44e6e00ac5672a59b25a56ecd80d2c2881 WatchSource:0}: Error finding container cb2e07972eb0eb7a84d75bc017091f44e6e00ac5672a59b25a56ecd80d2c2881: Status 404 returned error can't find the container with id cb2e07972eb0eb7a84d75bc017091f44e6e00ac5672a59b25a56ecd80d2c2881
Sep 18 16:30:00.593557 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.593534    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf"]
Sep 18 16:30:00.596444 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:30:00.596421    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2360b9a_50ee_488e_b34d_52a03308b0bc.slice/crio-fbbb3ccbae91fd43d7892c7a9aa7619cbdfaf1d38c17485c43789e5c67ebc9e2 WatchSource:0}: Error finding container fbbb3ccbae91fd43d7892c7a9aa7619cbdfaf1d38c17485c43789e5c67ebc9e2: Status 404 returned error can't find the container with id fbbb3ccbae91fd43d7892c7a9aa7619cbdfaf1d38c17485c43789e5c67ebc9e2
Sep 18 16:30:00.609829 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.609803    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829150-jsbcr"]
Sep 18 16:30:00.611716 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:30:00.611696    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ccf59d_3206_4435_9e2c_8766851cdf4f.slice/crio-293adba39c1dcf5a8faae7824883994dc7982197496dae6b1f84d41c6ab1a7e1 WatchSource:0}: Error finding container 293adba39c1dcf5a8faae7824883994dc7982197496dae6b1f84d41c6ab1a7e1: Status 404 returned error can't find the container with id 293adba39c1dcf5a8faae7824883994dc7982197496dae6b1f84d41c6ab1a7e1
Sep 18 16:30:00.636090 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.636057    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829150-djk5t" event={"ID":"c13899d7-4176-4980-88d9-93e3641f2950","Type":"ContainerStarted","Data":"cb2e07972eb0eb7a84d75bc017091f44e6e00ac5672a59b25a56ecd80d2c2881"}
Sep 18 16:30:00.637107 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.637086    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5865ea-29829150-6c4cr" event={"ID":"db097701-3bef-4ba8-b4a3-fd5fda8c3dc9","Type":"ContainerStarted","Data":"c938ca28696e8d80a384423f98a1a6b2a9c3112c8e5d3b67a4769f932277e4eb"}
Sep 18 16:30:00.638210 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.638187    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829150-jsbcr" event={"ID":"41ccf59d-3206-4435-9e2c-8766851cdf4f","Type":"ContainerStarted","Data":"293adba39c1dcf5a8faae7824883994dc7982197496dae6b1f84d41c6ab1a7e1"}
Sep 18 16:30:00.639200 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.639184    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf" event={"ID":"d2360b9a-50ee-488e-b34d-52a03308b0bc","Type":"ContainerStarted","Data":"fbbb3ccbae91fd43d7892c7a9aa7619cbdfaf1d38c17485c43789e5c67ebc9e2"}
Sep 18 16:30:00.729506 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:00.729473    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6" podUID="12f01686-2697-4d23-aed6-d4911b18ce37" containerName="main" containerID="cri-o://9d49de3a91742d343bba2ef04bdd249c4c7ff9c3c0aee98fa5084503e9449751" gracePeriod=45
Sep 18 16:30:01.055496 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.055472    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:30:01.187354 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.187324    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt7rq\" (UniqueName: \"kubernetes.io/projected/12f01686-2697-4d23-aed6-d4911b18ce37-kube-api-access-mt7rq\") pod \"12f01686-2697-4d23-aed6-d4911b18ce37\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") "
Sep 18 16:30:01.187785 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.187411    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-tmp-dir\") pod \"12f01686-2697-4d23-aed6-d4911b18ce37\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") "
Sep 18 16:30:01.187949 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.187456    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f01686-2697-4d23-aed6-d4911b18ce37-tls-certs\") pod \"12f01686-2697-4d23-aed6-d4911b18ce37\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") "
Sep 18 16:30:01.188044 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.188010    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-home\") pod \"12f01686-2697-4d23-aed6-d4911b18ce37\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") "
Sep 18 16:30:01.188106 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.188067    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-model-cache\") pod \"12f01686-2697-4d23-aed6-d4911b18ce37\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") "
Sep 18 16:30:01.188106 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.188085    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "12f01686-2697-4d23-aed6-d4911b18ce37" (UID: "12f01686-2697-4d23-aed6-d4911b18ce37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:30:01.188206 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.188189    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-dshm\") pod \"12f01686-2697-4d23-aed6-d4911b18ce37\" (UID: \"12f01686-2697-4d23-aed6-d4911b18ce37\") "
Sep 18 16:30:01.188329 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.188295    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-model-cache" (OuterVolumeSpecName: "model-cache") pod "12f01686-2697-4d23-aed6-d4911b18ce37" (UID: "12f01686-2697-4d23-aed6-d4911b18ce37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:30:01.188596 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.188568    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-home" (OuterVolumeSpecName: "home") pod "12f01686-2697-4d23-aed6-d4911b18ce37" (UID: "12f01686-2697-4d23-aed6-d4911b18ce37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:30:01.191082 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.189896    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:30:01.191082 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.189924    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:30:01.191082 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.189947    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:30:01.280808 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.280764    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12f01686-2697-4d23-aed6-d4911b18ce37-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "12f01686-2697-4d23-aed6-d4911b18ce37" (UID: "12f01686-2697-4d23-aed6-d4911b18ce37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:30:01.281629 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.281513    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f01686-2697-4d23-aed6-d4911b18ce37-kube-api-access-mt7rq" (OuterVolumeSpecName: "kube-api-access-mt7rq") pod "12f01686-2697-4d23-aed6-d4911b18ce37" (UID: "12f01686-2697-4d23-aed6-d4911b18ce37"). InnerVolumeSpecName "kube-api-access-mt7rq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:30:01.281629 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.281583    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-dshm" (OuterVolumeSpecName: "dshm") pod "12f01686-2697-4d23-aed6-d4911b18ce37" (UID: "12f01686-2697-4d23-aed6-d4911b18ce37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:30:01.290556 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.290531    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt7rq\" (UniqueName: \"kubernetes.io/projected/12f01686-2697-4d23-aed6-d4911b18ce37-kube-api-access-mt7rq\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:30:01.290656 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.290557    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12f01686-2697-4d23-aed6-d4911b18ce37-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:30:01.290656 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.290567    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12f01686-2697-4d23-aed6-d4911b18ce37-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:30:01.645971 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.645935    2546 generic.go:356] "Generic (PLEG): container finished" podID="12f01686-2697-4d23-aed6-d4911b18ce37" containerID="9d49de3a91742d343bba2ef04bdd249c4c7ff9c3c0aee98fa5084503e9449751" exitCode=0
Sep 18 16:30:01.646160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.645999    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"
Sep 18 16:30:01.646160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.646019    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6" event={"ID":"12f01686-2697-4d23-aed6-d4911b18ce37","Type":"ContainerDied","Data":"9d49de3a91742d343bba2ef04bdd249c4c7ff9c3c0aee98fa5084503e9449751"}
Sep 18 16:30:01.646160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.646066    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6" event={"ID":"12f01686-2697-4d23-aed6-d4911b18ce37","Type":"ContainerDied","Data":"2859fe22b5aa7b9fd057173857613605bfd43ed59c56a9934e76068b08ccc3e6"}
Sep 18 16:30:01.646160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.646086    2546 scope.go:117] "RemoveContainer" containerID="9d49de3a91742d343bba2ef04bdd249c4c7ff9c3c0aee98fa5084503e9449751"
Sep 18 16:30:01.648096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.648038    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829150-djk5t" event={"ID":"c13899d7-4176-4980-88d9-93e3641f2950","Type":"ContainerStarted","Data":"6527d982673fd2050c22f0904e0257a0c6da350bf221c53ba8b3c544c8b67511"}
Sep 18 16:30:01.649715 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.649695    2546 generic.go:356] "Generic (PLEG): container finished" podID="db097701-3bef-4ba8-b4a3-fd5fda8c3dc9" containerID="73aff6d7c104064980b7284f3ffb5e00a5f9a7e19b67478fff872f5c6a6f346a" exitCode=0
Sep 18 16:30:01.649833 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.649763    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5865ea-29829150-6c4cr" event={"ID":"db097701-3bef-4ba8-b4a3-fd5fda8c3dc9","Type":"ContainerDied","Data":"73aff6d7c104064980b7284f3ffb5e00a5f9a7e19b67478fff872f5c6a6f346a"}
Sep 18 16:30:01.651441 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.651422    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829150-jsbcr" event={"ID":"41ccf59d-3206-4435-9e2c-8766851cdf4f","Type":"ContainerStarted","Data":"85c8db09e4004fe1bae41016b331b4c1e5ec357d6b14a26a80c3d59f7ab90546"}
Sep 18 16:30:01.653069 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.653049    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf" event={"ID":"d2360b9a-50ee-488e-b34d-52a03308b0bc","Type":"ContainerStarted","Data":"865328039ec4b9c51fa3b71f186343d5b1ca728213c3b2f1d858375e5dfad8f2"}
Sep 18 16:30:01.656891 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.656877    2546 scope.go:117] "RemoveContainer" containerID="9d49de3a91742d343bba2ef04bdd249c4c7ff9c3c0aee98fa5084503e9449751"
Sep 18 16:30:01.657142 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:30:01.657125    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d49de3a91742d343bba2ef04bdd249c4c7ff9c3c0aee98fa5084503e9449751\": container with ID starting with 9d49de3a91742d343bba2ef04bdd249c4c7ff9c3c0aee98fa5084503e9449751 not found: ID does not exist" containerID="9d49de3a91742d343bba2ef04bdd249c4c7ff9c3c0aee98fa5084503e9449751"
Sep 18 16:30:01.657207 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.657148    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d49de3a91742d343bba2ef04bdd249c4c7ff9c3c0aee98fa5084503e9449751"} err="failed to get container status \"9d49de3a91742d343bba2ef04bdd249c4c7ff9c3c0aee98fa5084503e9449751\": rpc error: code = NotFound desc = could not find container \"9d49de3a91742d343bba2ef04bdd249c4c7ff9c3c0aee98fa5084503e9449751\": container with ID starting with 9d49de3a91742d343bba2ef04bdd249c4c7ff9c3c0aee98fa5084503e9449751 not found: ID does not exist"
Sep 18 16:30:01.704263 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.704221    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829150-jsbcr" podStartSLOduration=1.704208628 podStartE2EDuration="1.704208628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:30:01.702447308 +0000 UTC m=+1497.611227659" watchObservedRunningTime="2026-09-18 16:30:01.704208628 +0000 UTC m=+1497.612988978"
Sep 18 16:30:01.704392 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.704320    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829150-djk5t" podStartSLOduration=1.704315464 podStartE2EDuration="1.704315464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:30:01.68218893 +0000 UTC m=+1497.590969282" watchObservedRunningTime="2026-09-18 16:30:01.704315464 +0000 UTC m=+1497.613095856"
Sep 18 16:30:01.744588 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.744553    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"]
Sep 18 16:30:01.750740 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.750716    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-157a333a/e2e-lifecycle-model-157a333a-kserve-9787b6f6c-msmt6"]
Sep 18 16:30:01.773386 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:01.773331    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf" podStartSLOduration=1.773316902 podStartE2EDuration="1.773316902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:30:01.772265957 +0000 UTC m=+1497.681046307" watchObservedRunningTime="2026-09-18 16:30:01.773316902 +0000 UTC m=+1497.682097247"
Sep 18 16:30:02.603719 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:02.603685    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12f01686-2697-4d23-aed6-d4911b18ce37" path="/var/lib/kubelet/pods/12f01686-2697-4d23-aed6-d4911b18ce37/volumes"
Sep 18 16:30:02.661262 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:02.661237    2546 generic.go:356] "Generic (PLEG): container finished" podID="c13899d7-4176-4980-88d9-93e3641f2950" containerID="6527d982673fd2050c22f0904e0257a0c6da350bf221c53ba8b3c544c8b67511" exitCode=0
Sep 18 16:30:02.661412 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:02.661308    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829150-djk5t" event={"ID":"c13899d7-4176-4980-88d9-93e3641f2950","Type":"ContainerDied","Data":"6527d982673fd2050c22f0904e0257a0c6da350bf221c53ba8b3c544c8b67511"}
Sep 18 16:30:02.662800 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:02.662782    2546 generic.go:356] "Generic (PLEG): container finished" podID="41ccf59d-3206-4435-9e2c-8766851cdf4f" containerID="85c8db09e4004fe1bae41016b331b4c1e5ec357d6b14a26a80c3d59f7ab90546" exitCode=0
Sep 18 16:30:02.662871 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:02.662850    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829150-jsbcr" event={"ID":"41ccf59d-3206-4435-9e2c-8766851cdf4f","Type":"ContainerDied","Data":"85c8db09e4004fe1bae41016b331b4c1e5ec357d6b14a26a80c3d59f7ab90546"}
Sep 18 16:30:02.664356 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:02.664336    2546 generic.go:356] "Generic (PLEG): container finished" podID="d2360b9a-50ee-488e-b34d-52a03308b0bc" containerID="865328039ec4b9c51fa3b71f186343d5b1ca728213c3b2f1d858375e5dfad8f2" exitCode=0
Sep 18 16:30:02.664480 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:02.664361    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf" event={"ID":"d2360b9a-50ee-488e-b34d-52a03308b0bc","Type":"ContainerDied","Data":"865328039ec4b9c51fa3b71f186343d5b1ca728213c3b2f1d858375e5dfad8f2"}
Sep 18 16:30:02.813277 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:02.813249    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5865ea-29829150-6c4cr"
Sep 18 16:30:03.671772 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:03.671744    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5865ea-29829150-6c4cr"
Sep 18 16:30:03.672231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:03.671789    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5865ea-29829150-6c4cr" event={"ID":"db097701-3bef-4ba8-b4a3-fd5fda8c3dc9","Type":"ContainerDied","Data":"c938ca28696e8d80a384423f98a1a6b2a9c3112c8e5d3b67a4769f932277e4eb"}
Sep 18 16:30:03.672231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:03.671832    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c938ca28696e8d80a384423f98a1a6b2a9c3112c8e5d3b67a4769f932277e4eb"
Sep 18 16:30:03.766187 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:03.766049    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:30:03.832532 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:03.832508    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf"
Sep 18 16:30:03.867166 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:03.867139    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829150-jsbcr"
Sep 18 16:30:03.871539 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:03.871522    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829150-djk5t"
Sep 18 16:30:04.679959 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:04.679935    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829150-djk5t"
Sep 18 16:30:04.680861 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:04.679945    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829150-djk5t" event={"ID":"c13899d7-4176-4980-88d9-93e3641f2950","Type":"ContainerDied","Data":"cb2e07972eb0eb7a84d75bc017091f44e6e00ac5672a59b25a56ecd80d2c2881"}
Sep 18 16:30:04.681018 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:04.681005    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb2e07972eb0eb7a84d75bc017091f44e6e00ac5672a59b25a56ecd80d2c2881"
Sep 18 16:30:04.682892 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:04.682177    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829150-jsbcr"
Sep 18 16:30:04.682892 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:04.682547    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829150-jsbcr" event={"ID":"41ccf59d-3206-4435-9e2c-8766851cdf4f","Type":"ContainerDied","Data":"293adba39c1dcf5a8faae7824883994dc7982197496dae6b1f84d41c6ab1a7e1"}
Sep 18 16:30:04.682892 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:04.682588    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="293adba39c1dcf5a8faae7824883994dc7982197496dae6b1f84d41c6ab1a7e1"
Sep 18 16:30:04.684479 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:04.684450    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf" event={"ID":"d2360b9a-50ee-488e-b34d-52a03308b0bc","Type":"ContainerDied","Data":"fbbb3ccbae91fd43d7892c7a9aa7619cbdfaf1d38c17485c43789e5c67ebc9e2"}
Sep 18 16:30:04.684594 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:04.684501    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbbb3ccbae91fd43d7892c7a9aa7619cbdfaf1d38c17485c43789e5c67ebc9e2"
Sep 18 16:30:04.684594 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:04.684508    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf"
Sep 18 16:30:35.366923 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:35.366841    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:30:56.449728 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.449698    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"]
Sep 18 16:30:56.450196 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450146    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c13899d7-4176-4980-88d9-93e3641f2950" containerName="cleanup"
Sep 18 16:30:56.450196 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450163    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c13899d7-4176-4980-88d9-93e3641f2950" containerName="cleanup"
Sep 18 16:30:56.450196 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450174    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2360b9a-50ee-488e-b34d-52a03308b0bc" containerName="cleanup"
Sep 18 16:30:56.450196 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450179    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2360b9a-50ee-488e-b34d-52a03308b0bc" containerName="cleanup"
Sep 18 16:30:56.450196 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450195    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ccf59d-3206-4435-9e2c-8766851cdf4f" containerName="cleanup"
Sep 18 16:30:56.450360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450201    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ccf59d-3206-4435-9e2c-8766851cdf4f" containerName="cleanup"
Sep 18 16:30:56.450360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450211    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db097701-3bef-4ba8-b4a3-fd5fda8c3dc9" containerName="cleanup"
Sep 18 16:30:56.450360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450217    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="db097701-3bef-4ba8-b4a3-fd5fda8c3dc9" containerName="cleanup"
Sep 18 16:30:56.450360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450225    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f01686-2697-4d23-aed6-d4911b18ce37" containerName="main"
Sep 18 16:30:56.450360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450231    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f01686-2697-4d23-aed6-d4911b18ce37" containerName="main"
Sep 18 16:30:56.450360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450293    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ccf59d-3206-4435-9e2c-8766851cdf4f" containerName="cleanup"
Sep 18 16:30:56.450360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450301    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c13899d7-4176-4980-88d9-93e3641f2950" containerName="cleanup"
Sep 18 16:30:56.450360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450310    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="12f01686-2697-4d23-aed6-d4911b18ce37" containerName="main"
Sep 18 16:30:56.450360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450319    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2360b9a-50ee-488e-b34d-52a03308b0bc" containerName="cleanup"
Sep 18 16:30:56.450360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.450328    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="db097701-3bef-4ba8-b4a3-fd5fda8c3dc9" containerName="cleanup"
Sep 18 16:30:56.453506 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.453491    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.457647 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.457625    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4f2d8a\"/\"distinct-2-w4-4f2d8a-kserve-self-signed-certs\""
Sep 18 16:30:56.471344 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.471321    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"]
Sep 18 16:30:56.584250 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.584219    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc882f63-f895-454b-9e1d-51757fb7af58-tls-certs\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.584250 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.584255    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-home\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.584475 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.584360    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-tmp-dir\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.584475 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.584389    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppdzn\" (UniqueName: \"kubernetes.io/projected/fc882f63-f895-454b-9e1d-51757fb7af58-kube-api-access-ppdzn\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.584475 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.584467    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-dshm\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.584600 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.584560    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-model-cache\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.685328 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.685283    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-model-cache\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.685528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.685381    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc882f63-f895-454b-9e1d-51757fb7af58-tls-certs\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.685528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.685401    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-home\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.685528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.685449    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-tmp-dir\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.685528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.685481    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppdzn\" (UniqueName: \"kubernetes.io/projected/fc882f63-f895-454b-9e1d-51757fb7af58-kube-api-access-ppdzn\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.685758 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.685531    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-dshm\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.685830 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.685760    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-model-cache\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.685917 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.685874    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-tmp-dir\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.685973 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.685911    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-home\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.687805 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.687776    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-dshm\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.687912 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.687889    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc882f63-f895-454b-9e1d-51757fb7af58-tls-certs\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.698229 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.698206    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppdzn\" (UniqueName: \"kubernetes.io/projected/fc882f63-f895-454b-9e1d-51757fb7af58-kube-api-access-ppdzn\") pod \"distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") " pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:56.763149 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:56.763069    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:30:57.047722 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:57.047694    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"]
Sep 18 16:30:57.048764 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:30:57.048738    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc882f63_f895_454b_9e1d_51757fb7af58.slice/crio-a98e72df5672be92e6cccc67c0592a1f7990a57b7b143a424715ee839021f5e8 WatchSource:0}: Error finding container a98e72df5672be92e6cccc67c0592a1f7990a57b7b143a424715ee839021f5e8: Status 404 returned error can't find the container with id a98e72df5672be92e6cccc67c0592a1f7990a57b7b143a424715ee839021f5e8
Sep 18 16:30:58.017961 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:58.017917    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg" event={"ID":"fc882f63-f895-454b-9e1d-51757fb7af58","Type":"ContainerStarted","Data":"3ae4c01eb4172132e11975e4d23f82dc23530801a3f831e2e4935e3774dd3fc6"}
Sep 18 16:30:58.018350 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:58.017972    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg" event={"ID":"fc882f63-f895-454b-9e1d-51757fb7af58","Type":"ContainerStarted","Data":"a98e72df5672be92e6cccc67c0592a1f7990a57b7b143a424715ee839021f5e8"}
Sep 18 16:30:58.058704 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:30:58.058651    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg" podStartSLOduration=2.058637301 podStartE2EDuration="2.058637301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:30:58.058280536 +0000 UTC m=+1553.967060906" watchObservedRunningTime="2026-09-18 16:30:58.058637301 +0000 UTC m=+1553.967417652"
Sep 18 16:31:04.924381 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:31:04.924348    2546 scope.go:117] "RemoveContainer" containerID="2c8a6401bf16374a8320453036c9e4e4c0d4a4223f8a54196e1e3fceb09fe460"
Sep 18 16:31:04.933511 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:31:04.933484    2546 scope.go:117] "RemoveContainer" containerID="1c67273ed91e49b7ec71e243e6bc3f765334fe69eb68339eb19be0db0ae5a65e"
Sep 18 16:31:06.465964 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:31:06.465926    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:31:06.763808 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:31:06.763724    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:31:06.763985 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:31:06.763818    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:31:06.781562 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:31:06.781536    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:31:07.080520 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:31:07.080432    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:31:44.089692 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:31:44.089654    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:32:08.324372 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.324296    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"]
Sep 18 16:32:08.332902 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.332872    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.346970 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.346947    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9eae22-openshift-default-dockercfg-n9ghk\""
Sep 18 16:32:08.368739 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.368714    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"]
Sep 18 16:32:08.425386 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.421724    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-istio-envoy\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.425386 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.421799    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a30c401-5c7d-48bb-9005-60d688655187-istiod-ca-cert\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.425386 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.421882    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s8wj\" (UniqueName: \"kubernetes.io/projected/6a30c401-5c7d-48bb-9005-60d688655187-kube-api-access-7s8wj\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.425386 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.421930    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a30c401-5c7d-48bb-9005-60d688655187-istio-token\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.425386 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.421983    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a30c401-5c7d-48bb-9005-60d688655187-istio-podinfo\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.425386 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.422054    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-workload-socket\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.425386 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.422096    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-istio-data\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.425386 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.422126    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-credential-socket\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.425386 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.422153    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-workload-certs\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.523561 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.523530    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-workload-socket\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.523732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.523580    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-istio-data\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.523732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.523608    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-credential-socket\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.523732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.523628    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-workload-certs\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.523732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.523659    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-istio-envoy\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.523732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.523713    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a30c401-5c7d-48bb-9005-60d688655187-istiod-ca-cert\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.523970 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.523789    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s8wj\" (UniqueName: \"kubernetes.io/projected/6a30c401-5c7d-48bb-9005-60d688655187-kube-api-access-7s8wj\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.523970 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.523843    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a30c401-5c7d-48bb-9005-60d688655187-istio-token\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.523970 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.523895    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a30c401-5c7d-48bb-9005-60d688655187-istio-podinfo\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.523970 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.523922    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-workload-socket\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.523970 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.523960    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-credential-socket\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.524266 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.524077    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-workload-certs\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.524266 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.524107    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-istio-data\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.524525 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.524503    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a30c401-5c7d-48bb-9005-60d688655187-istiod-ca-cert\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.526214 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.526194    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-istio-envoy\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.526312 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.526296    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a30c401-5c7d-48bb-9005-60d688655187-istio-podinfo\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.534545 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.534522    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a30c401-5c7d-48bb-9005-60d688655187-istio-token\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.534750 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.534731    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s8wj\" (UniqueName: \"kubernetes.io/projected/6a30c401-5c7d-48bb-9005-60d688655187-kube-api-access-7s8wj\") pod \"e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") " pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.643498 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.643425    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:08.934253 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.934229    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"]
Sep 18 16:32:08.937118 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:32:08.937085    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a30c401_5c7d_48bb_9005_60d688655187.slice/crio-fe2e2f81d7e79103c08c93ec5ebe8614ba8ea78acb9787269d181b5415b4e875 WatchSource:0}: Error finding container fe2e2f81d7e79103c08c93ec5ebe8614ba8ea78acb9787269d181b5415b4e875: Status 404 returned error can't find the container with id fe2e2f81d7e79103c08c93ec5ebe8614ba8ea78acb9787269d181b5415b4e875
Sep 18 16:32:08.939633 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.939602    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:32:08.939739 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.939668    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:32:08.939739 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:08.939697    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 16:32:09.462958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:09.462923    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2" event={"ID":"6a30c401-5c7d-48bb-9005-60d688655187","Type":"ContainerStarted","Data":"944bc5de96b802761554c72eff7b0015a43ce55c686986af50e1ae386bb2fcaf"}
Sep 18 16:32:09.463422 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:09.462984    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2" event={"ID":"6a30c401-5c7d-48bb-9005-60d688655187","Type":"ContainerStarted","Data":"fe2e2f81d7e79103c08c93ec5ebe8614ba8ea78acb9787269d181b5415b4e875"}
Sep 18 16:32:09.503505 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:09.503457    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2" podStartSLOduration=1.503442105 podStartE2EDuration="1.503442105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:09.49856988 +0000 UTC m=+1625.407350251" watchObservedRunningTime="2026-09-18 16:32:09.503442105 +0000 UTC m=+1625.412222510"
Sep 18 16:32:09.644348 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:09.644314    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:09.645722 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:09.645697    2546 patch_prober.go:28] interesting pod/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 18 16:32:09.645845 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:09.645744    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2" podUID="6a30c401-5c7d-48bb-9005-60d688655187" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 18 16:32:10.648458 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:10.648430    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:11.474485 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:11.474449    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:11.475319 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:11.475299    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:32:20.812902 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:20.812871    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:32:29.154459 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:29.154423    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv"]
Sep 18 16:32:29.158159 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:29.158142    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv"
Sep 18 16:32:29.164153 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:29.164115    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:32:29.164287 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:29.164205    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:32:29.180647 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:29.180619    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv"]
Sep 18 16:32:29.254500 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:29.254451    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d48a9d5-4dc1-4e35-93d0-1817014fde5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv\" (UID: \"5d48a9d5-4dc1-4e35-93d0-1817014fde5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv"
Sep 18 16:32:29.355416 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:29.355377    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d48a9d5-4dc1-4e35-93d0-1817014fde5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv\" (UID: \"5d48a9d5-4dc1-4e35-93d0-1817014fde5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv"
Sep 18 16:32:29.355970 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:29.355950    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d48a9d5-4dc1-4e35-93d0-1817014fde5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv\" (UID: \"5d48a9d5-4dc1-4e35-93d0-1817014fde5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv"
Sep 18 16:32:29.468881 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:29.468845    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv"
Sep 18 16:32:29.759337 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:29.759312    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv"]
Sep 18 16:32:29.761903 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:32:29.761876    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d48a9d5_4dc1_4e35_93d0_1817014fde5b.slice/crio-6cdba7e1a8e61c07ca0805ccd08d9d27beec83155101449d0e33e029bfd06fb4 WatchSource:0}: Error finding container 6cdba7e1a8e61c07ca0805ccd08d9d27beec83155101449d0e33e029bfd06fb4: Status 404 returned error can't find the container with id 6cdba7e1a8e61c07ca0805ccd08d9d27beec83155101449d0e33e029bfd06fb4
Sep 18 16:32:30.590529 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:30.590494    2546 generic.go:356] "Generic (PLEG): container finished" podID="5d48a9d5-4dc1-4e35-93d0-1817014fde5b" containerID="3a5142d1429b851e5f67019df6d4c45a516d8cef431a084044cdf4059fe2a5ef" exitCode=0
Sep 18 16:32:30.590894 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:30.590576    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv" event={"ID":"5d48a9d5-4dc1-4e35-93d0-1817014fde5b","Type":"ContainerDied","Data":"3a5142d1429b851e5f67019df6d4c45a516d8cef431a084044cdf4059fe2a5ef"}
Sep 18 16:32:30.590894 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:30.590621    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv" event={"ID":"5d48a9d5-4dc1-4e35-93d0-1817014fde5b","Type":"ContainerStarted","Data":"6cdba7e1a8e61c07ca0805ccd08d9d27beec83155101449d0e33e029bfd06fb4"}
Sep 18 16:32:31.741831 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:31.741807    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv"
Sep 18 16:32:31.883856 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:31.883757    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d48a9d5-4dc1-4e35-93d0-1817014fde5b-maas-api-service-ca\") pod \"5d48a9d5-4dc1-4e35-93d0-1817014fde5b\" (UID: \"5d48a9d5-4dc1-4e35-93d0-1817014fde5b\") "
Sep 18 16:32:31.884135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:31.884113    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d48a9d5-4dc1-4e35-93d0-1817014fde5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d48a9d5-4dc1-4e35-93d0-1817014fde5b" (UID: "5d48a9d5-4dc1-4e35-93d0-1817014fde5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:32:31.985263 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:31.985227    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d48a9d5-4dc1-4e35-93d0-1817014fde5b-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:32:32.604409 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:32.604383    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv"
Sep 18 16:32:32.604573 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:32.604420    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv" event={"ID":"5d48a9d5-4dc1-4e35-93d0-1817014fde5b","Type":"ContainerDied","Data":"6cdba7e1a8e61c07ca0805ccd08d9d27beec83155101449d0e33e029bfd06fb4"}
Sep 18 16:32:32.604573 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:32.604465    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cdba7e1a8e61c07ca0805ccd08d9d27beec83155101449d0e33e029bfd06fb4"
Sep 18 16:32:35.688970 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.688934    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"]
Sep 18 16:32:35.689408 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.689364    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d48a9d5-4dc1-4e35-93d0-1817014fde5b" containerName="revoke-keys"
Sep 18 16:32:35.689408 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.689375    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d48a9d5-4dc1-4e35-93d0-1817014fde5b" containerName="revoke-keys"
Sep 18 16:32:35.689482 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.689454    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d48a9d5-4dc1-4e35-93d0-1817014fde5b" containerName="revoke-keys"
Sep 18 16:32:35.692515 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.692494    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.696406 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.696385    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1eec1d\"/\"embedding-w0-1eec1d-kserve-self-signed-certs\""
Sep 18 16:32:35.713902 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.713876    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"]
Sep 18 16:32:35.829467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.829428    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779f560a-59c4-4a20-a06e-bbaf33612479-tls-certs\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.829744 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.829724    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-dshm\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.829879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.829863    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-tmp-dir\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.830097 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.830077    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76xsl\" (UniqueName: \"kubernetes.io/projected/779f560a-59c4-4a20-a06e-bbaf33612479-kube-api-access-76xsl\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.830243 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.830227    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-home\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.830390 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.830375    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-model-cache\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.931496 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.931461    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76xsl\" (UniqueName: \"kubernetes.io/projected/779f560a-59c4-4a20-a06e-bbaf33612479-kube-api-access-76xsl\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.931694 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.931511    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-home\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.931694 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.931546    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-model-cache\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.931694 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.931593    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779f560a-59c4-4a20-a06e-bbaf33612479-tls-certs\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.931694 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.931629    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-dshm\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.931694 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.931650    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-tmp-dir\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.931968 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.931929    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-home\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.932060 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.932009    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-model-cache\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.932110 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.932068    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-tmp-dir\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.933808 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.933789    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-dshm\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.934083 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.934068    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779f560a-59c4-4a20-a06e-bbaf33612479-tls-certs\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:35.941355 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:35.941297    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76xsl\" (UniqueName: \"kubernetes.io/projected/779f560a-59c4-4a20-a06e-bbaf33612479-kube-api-access-76xsl\") pod \"embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") " pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:36.002153 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:36.002124    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:36.288240 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:36.288209    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"]
Sep 18 16:32:36.288898 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:32:36.288870    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod779f560a_59c4_4a20_a06e_bbaf33612479.slice/crio-55b1d8deaabd3098360945b5acf21dfd80c9002b81f4d1a9de3640db098bdc71 WatchSource:0}: Error finding container 55b1d8deaabd3098360945b5acf21dfd80c9002b81f4d1a9de3640db098bdc71: Status 404 returned error can't find the container with id 55b1d8deaabd3098360945b5acf21dfd80c9002b81f4d1a9de3640db098bdc71
Sep 18 16:32:36.628876 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:36.628790    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5" event={"ID":"779f560a-59c4-4a20-a06e-bbaf33612479","Type":"ContainerStarted","Data":"98c7362ff3e1624c991059a840d03e04ed7cd09d42f50971976bfbf397b76a81"}
Sep 18 16:32:36.628876 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:36.628830    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5" event={"ID":"779f560a-59c4-4a20-a06e-bbaf33612479","Type":"ContainerStarted","Data":"55b1d8deaabd3098360945b5acf21dfd80c9002b81f4d1a9de3640db098bdc71"}
Sep 18 16:32:36.651999 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:36.651948    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5" podStartSLOduration=1.65192952 podStartE2EDuration="1.65192952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:36.650940103 +0000 UTC m=+1652.559720447" watchObservedRunningTime="2026-09-18 16:32:36.65192952 +0000 UTC m=+1652.560709897"
Sep 18 16:32:43.451031 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:43.450998    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/healthz\": dial tcp 10.133.0.34:8081: connect: connection refused"
Sep 18 16:32:43.668874 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:43.668847    2546 generic.go:356] "Generic (PLEG): container finished" podID="1c53eeff-c125-49c6-ab26-182dd935f398" containerID="d7b66f6dcf15f0ee940b6003c3a51567213d18789e0c372e06cdd646e21ee788" exitCode=137
Sep 18 16:32:43.668989 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:43.668919    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" event={"ID":"1c53eeff-c125-49c6-ab26-182dd935f398","Type":"ContainerDied","Data":"d7b66f6dcf15f0ee940b6003c3a51567213d18789e0c372e06cdd646e21ee788"}
Sep 18 16:32:43.669242 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:43.669229    2546 scope.go:117] "RemoveContainer" containerID="d7b66f6dcf15f0ee940b6003c3a51567213d18789e0c372e06cdd646e21ee788"
Sep 18 16:32:44.676498 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:44.676464    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" event={"ID":"1c53eeff-c125-49c6-ab26-182dd935f398","Type":"ContainerStarted","Data":"df9864ed5a6bb54db9cd5a5cd3f4d5c4751b13a3865a1f7de0a0cfc58b6b2d0c"}
Sep 18 16:32:44.676951 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:44.676662    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:32:45.959484 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:45.959453    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv"]
Sep 18 16:32:45.969485 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:45.969460    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ac5968-8cvfv"]
Sep 18 16:32:46.002885 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:46.002853    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:46.003004 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:46.002902    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:46.015151 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:46.015129    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:46.603923 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:46.603890    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d48a9d5-4dc1-4e35-93d0-1817014fde5b" path="/var/lib/kubelet/pods/5d48a9d5-4dc1-4e35-93d0-1817014fde5b/volumes"
Sep 18 16:32:46.699646 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:46.699622    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:32:55.144214 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:55.144177    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm"]
Sep 18 16:32:55.149088 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:55.149071    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm"
Sep 18 16:32:55.152185 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:55.152165    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:32:55.152297 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:55.152222    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:32:55.165096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:55.165068    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm"]
Sep 18 16:32:55.218092 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:55.218060    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee1e0e0-3962-4231-9082-775f6295358e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm\" (UID: \"bee1e0e0-3962-4231-9082-775f6295358e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm"
Sep 18 16:32:55.319250 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:55.319213    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee1e0e0-3962-4231-9082-775f6295358e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm\" (UID: \"bee1e0e0-3962-4231-9082-775f6295358e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm"
Sep 18 16:32:55.319884 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:55.319857    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee1e0e0-3962-4231-9082-775f6295358e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm\" (UID: \"bee1e0e0-3962-4231-9082-775f6295358e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm"
Sep 18 16:32:55.459428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:55.459397    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm"
Sep 18 16:32:55.685059 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:55.685012    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:32:55.764743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:55.764714    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm"]
Sep 18 16:32:55.766216 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:32:55.766185    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee1e0e0_3962_4231_9082_775f6295358e.slice/crio-4860582a413dd1522e71757f6d9324a7ff0b776da021671a74d802cedc8402e8 WatchSource:0}: Error finding container 4860582a413dd1522e71757f6d9324a7ff0b776da021671a74d802cedc8402e8: Status 404 returned error can't find the container with id 4860582a413dd1522e71757f6d9324a7ff0b776da021671a74d802cedc8402e8
Sep 18 16:32:56.750358 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:56.750326    2546 generic.go:356] "Generic (PLEG): container finished" podID="bee1e0e0-3962-4231-9082-775f6295358e" containerID="248c8ada99764c09e340f92e5694d978b2b8011c932ad266dd074513468ff4e2" exitCode=0
Sep 18 16:32:56.750781 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:56.750406    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm" event={"ID":"bee1e0e0-3962-4231-9082-775f6295358e","Type":"ContainerDied","Data":"248c8ada99764c09e340f92e5694d978b2b8011c932ad266dd074513468ff4e2"}
Sep 18 16:32:56.750781 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:56.750449    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm" event={"ID":"bee1e0e0-3962-4231-9082-775f6295358e","Type":"ContainerStarted","Data":"4860582a413dd1522e71757f6d9324a7ff0b776da021671a74d802cedc8402e8"}
Sep 18 16:32:57.899734 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:57.899712    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm"
Sep 18 16:32:57.944176 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:57.944147    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee1e0e0-3962-4231-9082-775f6295358e-maas-api-service-ca\") pod \"bee1e0e0-3962-4231-9082-775f6295358e\" (UID: \"bee1e0e0-3962-4231-9082-775f6295358e\") "
Sep 18 16:32:57.944548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:57.944525    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee1e0e0-3962-4231-9082-775f6295358e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bee1e0e0-3962-4231-9082-775f6295358e" (UID: "bee1e0e0-3962-4231-9082-775f6295358e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:32:58.045625 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:58.045551    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee1e0e0-3962-4231-9082-775f6295358e-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:32:58.764858 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:58.764829    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm"
Sep 18 16:32:58.765051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:58.764868    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm" event={"ID":"bee1e0e0-3962-4231-9082-775f6295358e","Type":"ContainerDied","Data":"4860582a413dd1522e71757f6d9324a7ff0b776da021671a74d802cedc8402e8"}
Sep 18 16:32:58.765051 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:32:58.764918    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4860582a413dd1522e71757f6d9324a7ff0b776da021671a74d802cedc8402e8"
Sep 18 16:33:04.969359 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:04.969328    2546 scope.go:117] "RemoveContainer" containerID="81d3dd337536f89046edcd3845747d48e4879c30b8b780910b11f9a47da71eca"
Sep 18 16:33:10.410530 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:10.410495    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:33:11.500905 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:11.500872    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm"]
Sep 18 16:33:11.512335 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:11.512307    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9eae22-xvfgm"]
Sep 18 16:33:12.609994 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:12.609962    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bee1e0e0-3962-4231-9082-775f6295358e" path="/var/lib/kubelet/pods/bee1e0e0-3962-4231-9082-775f6295358e/volumes"
Sep 18 16:33:21.847881 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:21.847844    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"]
Sep 18 16:33:21.848322 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:21.848079    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2" podUID="6a30c401-5c7d-48bb-9005-60d688655187" containerName="istio-proxy" containerID="cri-o://944bc5de96b802761554c72eff7b0015a43ce55c686986af50e1ae386bb2fcaf" gracePeriod=30
Sep 18 16:33:21.931462 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:21.931431    2546 generic.go:356] "Generic (PLEG): container finished" podID="6a30c401-5c7d-48bb-9005-60d688655187" containerID="944bc5de96b802761554c72eff7b0015a43ce55c686986af50e1ae386bb2fcaf" exitCode=0
Sep 18 16:33:21.933122 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:21.931751    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2" event={"ID":"6a30c401-5c7d-48bb-9005-60d688655187","Type":"ContainerDied","Data":"944bc5de96b802761554c72eff7b0015a43ce55c686986af50e1ae386bb2fcaf"}
Sep 18 16:33:22.022700 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.022674    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:33:22.105282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.105205    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a30c401-5c7d-48bb-9005-60d688655187-istio-token\") pod \"6a30c401-5c7d-48bb-9005-60d688655187\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") "
Sep 18 16:33:22.105282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.105240    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-istio-envoy\") pod \"6a30c401-5c7d-48bb-9005-60d688655187\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") "
Sep 18 16:33:22.105482 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.105357    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a30c401-5c7d-48bb-9005-60d688655187-istio-podinfo\") pod \"6a30c401-5c7d-48bb-9005-60d688655187\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") "
Sep 18 16:33:22.105540 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.105502    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-istio-data\") pod \"6a30c401-5c7d-48bb-9005-60d688655187\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") "
Sep 18 16:33:22.105596 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.105561    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-credential-socket\") pod \"6a30c401-5c7d-48bb-9005-60d688655187\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") "
Sep 18 16:33:22.105648 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.105594    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-workload-certs\") pod \"6a30c401-5c7d-48bb-9005-60d688655187\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") "
Sep 18 16:33:22.105648 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.105624    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-workload-socket\") pod \"6a30c401-5c7d-48bb-9005-60d688655187\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") "
Sep 18 16:33:22.106500 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.105661    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a30c401-5c7d-48bb-9005-60d688655187-istiod-ca-cert\") pod \"6a30c401-5c7d-48bb-9005-60d688655187\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") "
Sep 18 16:33:22.106500 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.105700    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s8wj\" (UniqueName: \"kubernetes.io/projected/6a30c401-5c7d-48bb-9005-60d688655187-kube-api-access-7s8wj\") pod \"6a30c401-5c7d-48bb-9005-60d688655187\" (UID: \"6a30c401-5c7d-48bb-9005-60d688655187\") "
Sep 18 16:33:22.106724 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.106647    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6a30c401-5c7d-48bb-9005-60d688655187" (UID: "6a30c401-5c7d-48bb-9005-60d688655187"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:22.107135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.107003    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6a30c401-5c7d-48bb-9005-60d688655187" (UID: "6a30c401-5c7d-48bb-9005-60d688655187"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:22.107135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.107072    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-istio-data" (OuterVolumeSpecName: "istio-data") pod "6a30c401-5c7d-48bb-9005-60d688655187" (UID: "6a30c401-5c7d-48bb-9005-60d688655187"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:22.107319 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.107227    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6a30c401-5c7d-48bb-9005-60d688655187" (UID: "6a30c401-5c7d-48bb-9005-60d688655187"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:22.107319 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.107266    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a30c401-5c7d-48bb-9005-60d688655187-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6a30c401-5c7d-48bb-9005-60d688655187" (UID: "6a30c401-5c7d-48bb-9005-60d688655187"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:22.108077 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.108040    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6a30c401-5c7d-48bb-9005-60d688655187" (UID: "6a30c401-5c7d-48bb-9005-60d688655187"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:33:22.108077 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.108065    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a30c401-5c7d-48bb-9005-60d688655187-istio-token" (OuterVolumeSpecName: "istio-token") pod "6a30c401-5c7d-48bb-9005-60d688655187" (UID: "6a30c401-5c7d-48bb-9005-60d688655187"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:22.108601 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.108569    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a30c401-5c7d-48bb-9005-60d688655187-kube-api-access-7s8wj" (OuterVolumeSpecName: "kube-api-access-7s8wj") pod "6a30c401-5c7d-48bb-9005-60d688655187" (UID: "6a30c401-5c7d-48bb-9005-60d688655187"). InnerVolumeSpecName "kube-api-access-7s8wj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:33:22.109296 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.109261    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a30c401-5c7d-48bb-9005-60d688655187-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6a30c401-5c7d-48bb-9005-60d688655187" (UID: "6a30c401-5c7d-48bb-9005-60d688655187"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:33:22.206795 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.206764    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-credential-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:33:22.206795 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.206791    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-workload-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:33:22.206795 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.206801    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-workload-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:33:22.207096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.206810    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a30c401-5c7d-48bb-9005-60d688655187-istiod-ca-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:33:22.207096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.206819    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s8wj\" (UniqueName: \"kubernetes.io/projected/6a30c401-5c7d-48bb-9005-60d688655187-kube-api-access-7s8wj\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:33:22.207096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.206828    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a30c401-5c7d-48bb-9005-60d688655187-istio-token\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:33:22.207096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.206835    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-istio-envoy\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:33:22.207096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.206842    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a30c401-5c7d-48bb-9005-60d688655187-istio-podinfo\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:33:22.207096 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.206849    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a30c401-5c7d-48bb-9005-60d688655187-istio-data\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:33:22.283341 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.283310    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk"]
Sep 18 16:33:22.283858 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.283841    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a30c401-5c7d-48bb-9005-60d688655187" containerName="istio-proxy"
Sep 18 16:33:22.283858 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.283859    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a30c401-5c7d-48bb-9005-60d688655187" containerName="istio-proxy"
Sep 18 16:33:22.284000 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.283871    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee1e0e0-3962-4231-9082-775f6295358e" containerName="revoke-keys"
Sep 18 16:33:22.284000 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.283877    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee1e0e0-3962-4231-9082-775f6295358e" containerName="revoke-keys"
Sep 18 16:33:22.284000 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.283971    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a30c401-5c7d-48bb-9005-60d688655187" containerName="istio-proxy"
Sep 18 16:33:22.284000 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.283977    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee1e0e0-3962-4231-9082-775f6295358e" containerName="revoke-keys"
Sep 18 16:33:22.286244 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.286223    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk"
Sep 18 16:33:22.290681 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.290664    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:33:22.290978 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.290959    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:33:22.318834 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.318806    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk"]
Sep 18 16:33:22.408544 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.408457    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e321b8-c853-43d3-8b14-4f2f11aaa977-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk\" (UID: \"70e321b8-c853-43d3-8b14-4f2f11aaa977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk"
Sep 18 16:33:22.509242 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.509211    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e321b8-c853-43d3-8b14-4f2f11aaa977-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk\" (UID: \"70e321b8-c853-43d3-8b14-4f2f11aaa977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk"
Sep 18 16:33:22.509832 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.509805    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e321b8-c853-43d3-8b14-4f2f11aaa977-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk\" (UID: \"70e321b8-c853-43d3-8b14-4f2f11aaa977\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk"
Sep 18 16:33:22.597255 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.597221    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk"
Sep 18 16:33:22.913710 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.913688    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk"]
Sep 18 16:33:22.915603 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:33:22.915574    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e321b8_c853_43d3_8b14_4f2f11aaa977.slice/crio-fd8d7fdae797e6804f8eb7c36dd3af4860593c54751b89ee35e234d2fc04a393 WatchSource:0}: Error finding container fd8d7fdae797e6804f8eb7c36dd3af4860593c54751b89ee35e234d2fc04a393: Status 404 returned error can't find the container with id fd8d7fdae797e6804f8eb7c36dd3af4860593c54751b89ee35e234d2fc04a393
Sep 18 16:33:22.938438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.938414    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk" event={"ID":"70e321b8-c853-43d3-8b14-4f2f11aaa977","Type":"ContainerStarted","Data":"fd8d7fdae797e6804f8eb7c36dd3af4860593c54751b89ee35e234d2fc04a393"}
Sep 18 16:33:22.939640 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.939623    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2" event={"ID":"6a30c401-5c7d-48bb-9005-60d688655187","Type":"ContainerDied","Data":"fe2e2f81d7e79103c08c93ec5ebe8614ba8ea78acb9787269d181b5415b4e875"}
Sep 18 16:33:22.939741 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.939674    2546 scope.go:117] "RemoveContainer" containerID="944bc5de96b802761554c72eff7b0015a43ce55c686986af50e1ae386bb2fcaf"
Sep 18 16:33:22.939811 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.939675    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"
Sep 18 16:33:22.993275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:22.993239    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"]
Sep 18 16:33:23.003292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:23.003267    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9eae22-openshift-default-78548b7f46-4mwf2"]
Sep 18 16:33:23.946633 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:23.946607    2546 generic.go:356] "Generic (PLEG): container finished" podID="70e321b8-c853-43d3-8b14-4f2f11aaa977" containerID="6d7f242db4d22d4ca9e80c02e4c743b34d288b370b15b4c34217585428f68ad6" exitCode=0
Sep 18 16:33:23.947079 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:23.946690    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk" event={"ID":"70e321b8-c853-43d3-8b14-4f2f11aaa977","Type":"ContainerDied","Data":"6d7f242db4d22d4ca9e80c02e4c743b34d288b370b15b4c34217585428f68ad6"}
Sep 18 16:33:24.603905 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:24.603872    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a30c401-5c7d-48bb-9005-60d688655187" path="/var/lib/kubelet/pods/6a30c401-5c7d-48bb-9005-60d688655187/volumes"
Sep 18 16:33:25.091469 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:25.091443    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk"
Sep 18 16:33:25.239987 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:25.239900    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e321b8-c853-43d3-8b14-4f2f11aaa977-maas-api-service-ca\") pod \"70e321b8-c853-43d3-8b14-4f2f11aaa977\" (UID: \"70e321b8-c853-43d3-8b14-4f2f11aaa977\") "
Sep 18 16:33:25.240322 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:25.240299    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e321b8-c853-43d3-8b14-4f2f11aaa977-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70e321b8-c853-43d3-8b14-4f2f11aaa977" (UID: "70e321b8-c853-43d3-8b14-4f2f11aaa977"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:25.341235 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:25.341197    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70e321b8-c853-43d3-8b14-4f2f11aaa977-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:33:25.961467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:25.961440    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk"
Sep 18 16:33:25.961637 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:25.961470    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk" event={"ID":"70e321b8-c853-43d3-8b14-4f2f11aaa977","Type":"ContainerDied","Data":"fd8d7fdae797e6804f8eb7c36dd3af4860593c54751b89ee35e234d2fc04a393"}
Sep 18 16:33:25.961637 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:25.961516    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd8d7fdae797e6804f8eb7c36dd3af4860593c54751b89ee35e234d2fc04a393"
Sep 18 16:33:33.014482 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:33.014403    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk"]
Sep 18 16:33:33.021923 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:33.021889    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5b98dc-cvvfk"]
Sep 18 16:33:34.603602 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:34.603570    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70e321b8-c853-43d3-8b14-4f2f11aaa977" path="/var/lib/kubelet/pods/70e321b8-c853-43d3-8b14-4f2f11aaa977/volumes"
Sep 18 16:33:42.212453 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.212417    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs"]
Sep 18 16:33:42.213054 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.213013    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70e321b8-c853-43d3-8b14-4f2f11aaa977" containerName="revoke-keys"
Sep 18 16:33:42.213054 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.213054    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="70e321b8-c853-43d3-8b14-4f2f11aaa977" containerName="revoke-keys"
Sep 18 16:33:42.213215 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.213146    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="70e321b8-c853-43d3-8b14-4f2f11aaa977" containerName="revoke-keys"
Sep 18 16:33:42.216428 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.216410    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs"
Sep 18 16:33:42.219943 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.219923    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:33:42.220087 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.219981    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:33:42.227409 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.227385    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs"]
Sep 18 16:33:42.323121 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.323092    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea1dbef-d03f-4060-98b8-a919be1bb44b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs\" (UID: \"6ea1dbef-d03f-4060-98b8-a919be1bb44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs"
Sep 18 16:33:42.424569 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.424527    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea1dbef-d03f-4060-98b8-a919be1bb44b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs\" (UID: \"6ea1dbef-d03f-4060-98b8-a919be1bb44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs"
Sep 18 16:33:42.425184 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.425167    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea1dbef-d03f-4060-98b8-a919be1bb44b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs\" (UID: \"6ea1dbef-d03f-4060-98b8-a919be1bb44b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs"
Sep 18 16:33:42.503507 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.503424    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc"]
Sep 18 16:33:42.506954 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.506939    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc"
Sep 18 16:33:42.518460 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.518430    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc"]
Sep 18 16:33:42.526639 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.526617    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs"
Sep 18 16:33:42.627406 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.627366    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b919690b-c250-4b19-824c-075c74e8439e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc\" (UID: \"b919690b-c250-4b19-824c-075c74e8439e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc"
Sep 18 16:33:42.728365 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.728328    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b919690b-c250-4b19-824c-075c74e8439e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc\" (UID: \"b919690b-c250-4b19-824c-075c74e8439e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc"
Sep 18 16:33:42.728937 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.728915    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b919690b-c250-4b19-824c-075c74e8439e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc\" (UID: \"b919690b-c250-4b19-824c-075c74e8439e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc"
Sep 18 16:33:42.810094 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.810066    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs"]
Sep 18 16:33:42.811990 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:33:42.811963    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea1dbef_d03f_4060_98b8_a919be1bb44b.slice/crio-b43fc83d4e62a84df9dc042b774b2800110ae65478512ff3bd75f559e42baaef WatchSource:0}: Error finding container b43fc83d4e62a84df9dc042b774b2800110ae65478512ff3bd75f559e42baaef: Status 404 returned error can't find the container with id b43fc83d4e62a84df9dc042b774b2800110ae65478512ff3bd75f559e42baaef
Sep 18 16:33:42.817117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:42.817098    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc"
Sep 18 16:33:43.068861 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:43.068778    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs" event={"ID":"6ea1dbef-d03f-4060-98b8-a919be1bb44b","Type":"ContainerStarted","Data":"b43fc83d4e62a84df9dc042b774b2800110ae65478512ff3bd75f559e42baaef"}
Sep 18 16:33:43.097199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:43.097175    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc"]
Sep 18 16:33:43.098783 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:33:43.098758    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb919690b_c250_4b19_824c_075c74e8439e.slice/crio-6d274e43279da4efb22bd53319cb9959a8748cbd91c33af28e5dee33a3026faf WatchSource:0}: Error finding container 6d274e43279da4efb22bd53319cb9959a8748cbd91c33af28e5dee33a3026faf: Status 404 returned error can't find the container with id 6d274e43279da4efb22bd53319cb9959a8748cbd91c33af28e5dee33a3026faf
Sep 18 16:33:44.076616 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:44.076532    2546 generic.go:356] "Generic (PLEG): container finished" podID="6ea1dbef-d03f-4060-98b8-a919be1bb44b" containerID="be7015b02006302cd4856bd647e5999f7111a5ef9f83d9e9a71ea9639490aaba" exitCode=0
Sep 18 16:33:44.076962 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:44.076617    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs" event={"ID":"6ea1dbef-d03f-4060-98b8-a919be1bb44b","Type":"ContainerDied","Data":"be7015b02006302cd4856bd647e5999f7111a5ef9f83d9e9a71ea9639490aaba"}
Sep 18 16:33:44.078297 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:44.078276    2546 generic.go:356] "Generic (PLEG): container finished" podID="b919690b-c250-4b19-824c-075c74e8439e" containerID="f360032c042b727c15af3dd440cbc8b5c3d01cdd1ddbc6e8b4b5906053e217cd" exitCode=0
Sep 18 16:33:44.078434 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:44.078356    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc" event={"ID":"b919690b-c250-4b19-824c-075c74e8439e","Type":"ContainerDied","Data":"f360032c042b727c15af3dd440cbc8b5c3d01cdd1ddbc6e8b4b5906053e217cd"}
Sep 18 16:33:44.078434 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:44.078399    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc" event={"ID":"b919690b-c250-4b19-824c-075c74e8439e","Type":"ContainerStarted","Data":"6d274e43279da4efb22bd53319cb9959a8748cbd91c33af28e5dee33a3026faf"}
Sep 18 16:33:45.261139 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:45.261120    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc"
Sep 18 16:33:45.264235 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:45.264216    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs"
Sep 18 16:33:45.359409 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:45.359328    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b919690b-c250-4b19-824c-075c74e8439e-maas-api-service-ca\") pod \"b919690b-c250-4b19-824c-075c74e8439e\" (UID: \"b919690b-c250-4b19-824c-075c74e8439e\") "
Sep 18 16:33:45.359558 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:45.359433    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea1dbef-d03f-4060-98b8-a919be1bb44b-maas-api-service-ca\") pod \"6ea1dbef-d03f-4060-98b8-a919be1bb44b\" (UID: \"6ea1dbef-d03f-4060-98b8-a919be1bb44b\") "
Sep 18 16:33:45.359709 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:45.359687    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b919690b-c250-4b19-824c-075c74e8439e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b919690b-c250-4b19-824c-075c74e8439e" (UID: "b919690b-c250-4b19-824c-075c74e8439e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:45.359773 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:45.359756    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea1dbef-d03f-4060-98b8-a919be1bb44b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ea1dbef-d03f-4060-98b8-a919be1bb44b" (UID: "6ea1dbef-d03f-4060-98b8-a919be1bb44b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:33:45.359810 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:45.359803    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b919690b-c250-4b19-824c-075c74e8439e-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:33:45.461241 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:45.461210    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ea1dbef-d03f-4060-98b8-a919be1bb44b-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:33:46.092470 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:46.092442    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs"
Sep 18 16:33:46.092470 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:46.092461    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs" event={"ID":"6ea1dbef-d03f-4060-98b8-a919be1bb44b","Type":"ContainerDied","Data":"b43fc83d4e62a84df9dc042b774b2800110ae65478512ff3bd75f559e42baaef"}
Sep 18 16:33:46.092710 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:46.092515    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b43fc83d4e62a84df9dc042b774b2800110ae65478512ff3bd75f559e42baaef"
Sep 18 16:33:46.093965 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:46.093935    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc"
Sep 18 16:33:46.093965 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:46.093941    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc" event={"ID":"b919690b-c250-4b19-824c-075c74e8439e","Type":"ContainerDied","Data":"6d274e43279da4efb22bd53319cb9959a8748cbd91c33af28e5dee33a3026faf"}
Sep 18 16:33:46.094190 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:46.093991    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d274e43279da4efb22bd53319cb9959a8748cbd91c33af28e5dee33a3026faf"
Sep 18 16:33:46.898859 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:46.898826    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:33:47.351053 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:47.350990    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"]
Sep 18 16:33:47.373460 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:47.373432    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"]
Sep 18 16:33:47.410248 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:47.410218    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"]
Sep 18 16:33:47.451406 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:47.451378    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"]
Sep 18 16:33:47.459732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:47.459702    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"]
Sep 18 16:33:53.111980 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:53.111932    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5865ea-29829150-6c4cr"]
Sep 18 16:33:53.114865 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:53.114838    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5865ea-29829150-6c4cr"]
Sep 18 16:33:53.624994 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:53.624956    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs"]
Sep 18 16:33:53.637220 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:53.637188    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5865ea-mfsrs"]
Sep 18 16:33:54.603679 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:54.603645    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ea1dbef-d03f-4060-98b8-a919be1bb44b" path="/var/lib/kubelet/pods/6ea1dbef-d03f-4060-98b8-a919be1bb44b/volumes"
Sep 18 16:33:54.604183 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:54.604133    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db097701-3bef-4ba8-b4a3-fd5fda8c3dc9" path="/var/lib/kubelet/pods/db097701-3bef-4ba8-b4a3-fd5fda8c3dc9/volumes"
Sep 18 16:33:55.999998 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:55.999965    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc"]
Sep 18 16:33:56.007062 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:56.007036    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-1eec1d-lr8lc"]
Sep 18 16:33:56.604799 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:33:56.604764    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b919690b-c250-4b19-824c-075c74e8439e" path="/var/lib/kubelet/pods/b919690b-c250-4b19-824c-075c74e8439e/volumes"
Sep 18 16:34:02.370893 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.370840    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt" podUID="f2691e4c-7d42-4c19-8e31-b32677395c3b" containerName="main" containerID="cri-o://e4e26a19242e67118f4fd252dc9379e33eb12347c454a6821ba929d9851cdfd3" gracePeriod=45
Sep 18 16:34:02.392100 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.392061    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j" podUID="4431196d-8458-4694-a9f5-6831131c6467" containerName="main" containerID="cri-o://8eadd26e5eb2221f892091f1d620980a24fdd33f750bbc11ffabdc17562367a9" gracePeriod=45
Sep 18 16:34:02.429668 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.429610    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5" podUID="779f560a-59c4-4a20-a06e-bbaf33612479" containerName="main" containerID="cri-o://98c7362ff3e1624c991059a840d03e04ed7cd09d42f50971976bfbf397b76a81" gracePeriod=45
Sep 18 16:34:02.472417 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.472378    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n" podUID="d7318181-8bda-493f-bc63-0bf78a370280" containerName="main" containerID="cri-o://16df1713faa603163b9b08e792d2b1532a50de1ec92fdf74eae3ed9c0fa67821" gracePeriod=45
Sep 18 16:34:02.478968 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.478916    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp" podUID="97bd1126-9a3e-400f-8350-733958d40f2e" containerName="main" containerID="cri-o://964342e43f63a572d951445f1b43a82f1928635b40830eab9760d17b0d6b4f2d" gracePeriod=45
Sep 18 16:34:02.703267 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.703233    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:34:02.774708 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.774628    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-home\") pod \"f2691e4c-7d42-4c19-8e31-b32677395c3b\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") "
Sep 18 16:34:02.774708 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.774665    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbktw\" (UniqueName: \"kubernetes.io/projected/f2691e4c-7d42-4c19-8e31-b32677395c3b-kube-api-access-fbktw\") pod \"f2691e4c-7d42-4c19-8e31-b32677395c3b\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") "
Sep 18 16:34:02.775807 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.774779    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-tmp-dir\") pod \"f2691e4c-7d42-4c19-8e31-b32677395c3b\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") "
Sep 18 16:34:02.775807 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.774881    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-model-cache\") pod \"f2691e4c-7d42-4c19-8e31-b32677395c3b\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") "
Sep 18 16:34:02.775807 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.774919    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-dshm\") pod \"f2691e4c-7d42-4c19-8e31-b32677395c3b\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") "
Sep 18 16:34:02.775807 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.774953    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2691e4c-7d42-4c19-8e31-b32677395c3b-tls-certs\") pod \"f2691e4c-7d42-4c19-8e31-b32677395c3b\" (UID: \"f2691e4c-7d42-4c19-8e31-b32677395c3b\") "
Sep 18 16:34:02.775807 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.775312    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2691e4c-7d42-4c19-8e31-b32677395c3b" (UID: "f2691e4c-7d42-4c19-8e31-b32677395c3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.776326 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.776240    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.785979 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.776596    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-home" (OuterVolumeSpecName: "home") pod "f2691e4c-7d42-4c19-8e31-b32677395c3b" (UID: "f2691e4c-7d42-4c19-8e31-b32677395c3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.785979 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.776676    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2691e4c-7d42-4c19-8e31-b32677395c3b" (UID: "f2691e4c-7d42-4c19-8e31-b32677395c3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.785979 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.780742    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2691e4c-7d42-4c19-8e31-b32677395c3b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2691e4c-7d42-4c19-8e31-b32677395c3b" (UID: "f2691e4c-7d42-4c19-8e31-b32677395c3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:02.785979 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.785644    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2691e4c-7d42-4c19-8e31-b32677395c3b-kube-api-access-fbktw" (OuterVolumeSpecName: "kube-api-access-fbktw") pod "f2691e4c-7d42-4c19-8e31-b32677395c3b" (UID: "f2691e4c-7d42-4c19-8e31-b32677395c3b"). InnerVolumeSpecName "kube-api-access-fbktw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:02.788766 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.788725    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-dshm" (OuterVolumeSpecName: "dshm") pod "f2691e4c-7d42-4c19-8e31-b32677395c3b" (UID: "f2691e4c-7d42-4c19-8e31-b32677395c3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.831216 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.830960    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:34:02.833314 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.833288    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:34:02.842636 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.842609    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:34:02.844967 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.844945    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:34:02.876815 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.876782    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-tmp-dir\") pod \"4431196d-8458-4694-a9f5-6831131c6467\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") "
Sep 18 16:34:02.876815 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.876820    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-model-cache\") pod \"d7318181-8bda-493f-bc63-0bf78a370280\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") "
Sep 18 16:34:02.877141 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.876844    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76xsl\" (UniqueName: \"kubernetes.io/projected/779f560a-59c4-4a20-a06e-bbaf33612479-kube-api-access-76xsl\") pod \"779f560a-59c4-4a20-a06e-bbaf33612479\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") "
Sep 18 16:34:02.877141 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.876869    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-home\") pod \"4431196d-8458-4694-a9f5-6831131c6467\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") "
Sep 18 16:34:02.877141 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877116    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4431196d-8458-4694-a9f5-6831131c6467" (UID: "4431196d-8458-4694-a9f5-6831131c6467"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.877141 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877129    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-home" (OuterVolumeSpecName: "home") pod "4431196d-8458-4694-a9f5-6831131c6467" (UID: "4431196d-8458-4694-a9f5-6831131c6467"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.877374 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877172    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-tmp-dir\") pod \"779f560a-59c4-4a20-a06e-bbaf33612479\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") "
Sep 18 16:34:02.877374 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877201    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-dshm\") pod \"4431196d-8458-4694-a9f5-6831131c6467\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") "
Sep 18 16:34:02.877374 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877207    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-model-cache" (OuterVolumeSpecName: "model-cache") pod "d7318181-8bda-493f-bc63-0bf78a370280" (UID: "d7318181-8bda-493f-bc63-0bf78a370280"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.877374 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877238    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-tmp-dir\") pod \"d7318181-8bda-493f-bc63-0bf78a370280\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") "
Sep 18 16:34:02.877374 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877300    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4431196d-8458-4694-a9f5-6831131c6467-tls-certs\") pod \"4431196d-8458-4694-a9f5-6831131c6467\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") "
Sep 18 16:34:02.877634 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877410    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "779f560a-59c4-4a20-a06e-bbaf33612479" (UID: "779f560a-59c4-4a20-a06e-bbaf33612479"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.877634 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877535    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d7318181-8bda-493f-bc63-0bf78a370280" (UID: "d7318181-8bda-493f-bc63-0bf78a370280"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.877634 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877604    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-dshm\") pod \"779f560a-59c4-4a20-a06e-bbaf33612479\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") "
Sep 18 16:34:02.877634 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877631    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-model-cache\") pod \"779f560a-59c4-4a20-a06e-bbaf33612479\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") "
Sep 18 16:34:02.877863 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877666    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljd4k\" (UniqueName: \"kubernetes.io/projected/4431196d-8458-4694-a9f5-6831131c6467-kube-api-access-ljd4k\") pod \"4431196d-8458-4694-a9f5-6831131c6467\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") "
Sep 18 16:34:02.877863 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877712    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-home\") pod \"d7318181-8bda-493f-bc63-0bf78a370280\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") "
Sep 18 16:34:02.877863 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877770    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-home\") pod \"779f560a-59c4-4a20-a06e-bbaf33612479\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") "
Sep 18 16:34:02.877863 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877797    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd8kg\" (UniqueName: \"kubernetes.io/projected/97bd1126-9a3e-400f-8350-733958d40f2e-kube-api-access-dd8kg\") pod \"97bd1126-9a3e-400f-8350-733958d40f2e\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") "
Sep 18 16:34:02.877863 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877838    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-dshm\") pod \"97bd1126-9a3e-400f-8350-733958d40f2e\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") "
Sep 18 16:34:02.878282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877889    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7318181-8bda-493f-bc63-0bf78a370280-tls-certs\") pod \"d7318181-8bda-493f-bc63-0bf78a370280\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") "
Sep 18 16:34:02.878282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877914    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-model-cache" (OuterVolumeSpecName: "model-cache") pod "779f560a-59c4-4a20-a06e-bbaf33612479" (UID: "779f560a-59c4-4a20-a06e-bbaf33612479"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.878282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877929    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-dshm\") pod \"d7318181-8bda-493f-bc63-0bf78a370280\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") "
Sep 18 16:34:02.878282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.877971    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-tmp-dir\") pod \"97bd1126-9a3e-400f-8350-733958d40f2e\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") "
Sep 18 16:34:02.878282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.878007    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bd1126-9a3e-400f-8350-733958d40f2e-tls-certs\") pod \"97bd1126-9a3e-400f-8350-733958d40f2e\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") "
Sep 18 16:34:02.878282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.878046    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779f560a-59c4-4a20-a06e-bbaf33612479-tls-certs\") pod \"779f560a-59c4-4a20-a06e-bbaf33612479\" (UID: \"779f560a-59c4-4a20-a06e-bbaf33612479\") "
Sep 18 16:34:02.878282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.878072    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-model-cache\") pod \"4431196d-8458-4694-a9f5-6831131c6467\" (UID: \"4431196d-8458-4694-a9f5-6831131c6467\") "
Sep 18 16:34:02.878282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.878101    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-model-cache\") pod \"97bd1126-9a3e-400f-8350-733958d40f2e\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") "
Sep 18 16:34:02.878282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.878127    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-home\") pod \"97bd1126-9a3e-400f-8350-733958d40f2e\" (UID: \"97bd1126-9a3e-400f-8350-733958d40f2e\") "
Sep 18 16:34:02.878282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.878167    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw5cx\" (UniqueName: \"kubernetes.io/projected/d7318181-8bda-493f-bc63-0bf78a370280-kube-api-access-qw5cx\") pod \"d7318181-8bda-493f-bc63-0bf78a370280\" (UID: \"d7318181-8bda-493f-bc63-0bf78a370280\") "
Sep 18 16:34:02.879528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.878373    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-home" (OuterVolumeSpecName: "home") pod "779f560a-59c4-4a20-a06e-bbaf33612479" (UID: "779f560a-59c4-4a20-a06e-bbaf33612479"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.879528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.879004    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.879528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.879053    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbktw\" (UniqueName: \"kubernetes.io/projected/f2691e4c-7d42-4c19-8e31-b32677395c3b-kube-api-access-fbktw\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.879528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.879071    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.879528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.879086    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.879528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.879099    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.879528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.879110    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2691e4c-7d42-4c19-8e31-b32677395c3b-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.879528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.879121    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.879528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.879134    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.879528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.879148    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2691e4c-7d42-4c19-8e31-b32677395c3b-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.879528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.879160    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.879528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.879172    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.879528 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.879184    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.884080 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.882851    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779f560a-59c4-4a20-a06e-bbaf33612479-kube-api-access-76xsl" (OuterVolumeSpecName: "kube-api-access-76xsl") pod "779f560a-59c4-4a20-a06e-bbaf33612479" (UID: "779f560a-59c4-4a20-a06e-bbaf33612479"). InnerVolumeSpecName "kube-api-access-76xsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:02.887927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.887644    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4431196d-8458-4694-a9f5-6831131c6467-kube-api-access-ljd4k" (OuterVolumeSpecName: "kube-api-access-ljd4k") pod "4431196d-8458-4694-a9f5-6831131c6467" (UID: "4431196d-8458-4694-a9f5-6831131c6467"). InnerVolumeSpecName "kube-api-access-ljd4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:02.887927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.887870    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-home" (OuterVolumeSpecName: "home") pod "d7318181-8bda-493f-bc63-0bf78a370280" (UID: "d7318181-8bda-493f-bc63-0bf78a370280"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.888408 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.888354    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97bd1126-9a3e-400f-8350-733958d40f2e" (UID: "97bd1126-9a3e-400f-8350-733958d40f2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.889106 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.889074    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-model-cache" (OuterVolumeSpecName: "model-cache") pod "4431196d-8458-4694-a9f5-6831131c6467" (UID: "4431196d-8458-4694-a9f5-6831131c6467"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.889371 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.889349    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-dshm" (OuterVolumeSpecName: "dshm") pod "d7318181-8bda-493f-bc63-0bf78a370280" (UID: "d7318181-8bda-493f-bc63-0bf78a370280"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.889901 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.889620    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-dshm" (OuterVolumeSpecName: "dshm") pod "4431196d-8458-4694-a9f5-6831131c6467" (UID: "4431196d-8458-4694-a9f5-6831131c6467"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.889901 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.889696    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7318181-8bda-493f-bc63-0bf78a370280-kube-api-access-qw5cx" (OuterVolumeSpecName: "kube-api-access-qw5cx") pod "d7318181-8bda-493f-bc63-0bf78a370280" (UID: "d7318181-8bda-493f-bc63-0bf78a370280"). InnerVolumeSpecName "kube-api-access-qw5cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:02.890345 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.890185    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-model-cache" (OuterVolumeSpecName: "model-cache") pod "97bd1126-9a3e-400f-8350-733958d40f2e" (UID: "97bd1126-9a3e-400f-8350-733958d40f2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.890548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.890525    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-home" (OuterVolumeSpecName: "home") pod "97bd1126-9a3e-400f-8350-733958d40f2e" (UID: "97bd1126-9a3e-400f-8350-733958d40f2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.890955 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.890932    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97bd1126-9a3e-400f-8350-733958d40f2e-kube-api-access-dd8kg" (OuterVolumeSpecName: "kube-api-access-dd8kg") pod "97bd1126-9a3e-400f-8350-733958d40f2e" (UID: "97bd1126-9a3e-400f-8350-733958d40f2e"). InnerVolumeSpecName "kube-api-access-dd8kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:02.891927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.891839    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4431196d-8458-4694-a9f5-6831131c6467-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4431196d-8458-4694-a9f5-6831131c6467" (UID: "4431196d-8458-4694-a9f5-6831131c6467"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:02.891927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.891875    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bd1126-9a3e-400f-8350-733958d40f2e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97bd1126-9a3e-400f-8350-733958d40f2e" (UID: "97bd1126-9a3e-400f-8350-733958d40f2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:02.891927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.891913    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-dshm" (OuterVolumeSpecName: "dshm") pod "779f560a-59c4-4a20-a06e-bbaf33612479" (UID: "779f560a-59c4-4a20-a06e-bbaf33612479"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.892479 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.892458    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-dshm" (OuterVolumeSpecName: "dshm") pod "97bd1126-9a3e-400f-8350-733958d40f2e" (UID: "97bd1126-9a3e-400f-8350-733958d40f2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:02.893013 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.892989    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7318181-8bda-493f-bc63-0bf78a370280-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d7318181-8bda-493f-bc63-0bf78a370280" (UID: "d7318181-8bda-493f-bc63-0bf78a370280"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:02.893482 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.893457    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779f560a-59c4-4a20-a06e-bbaf33612479-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "779f560a-59c4-4a20-a06e-bbaf33612479" (UID: "779f560a-59c4-4a20-a06e-bbaf33612479"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:02.980753 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980719    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4431196d-8458-4694-a9f5-6831131c6467-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.980753 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980752    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779f560a-59c4-4a20-a06e-bbaf33612479-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980767    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljd4k\" (UniqueName: \"kubernetes.io/projected/4431196d-8458-4694-a9f5-6831131c6467-kube-api-access-ljd4k\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980782    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980797    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd8kg\" (UniqueName: \"kubernetes.io/projected/97bd1126-9a3e-400f-8350-733958d40f2e-kube-api-access-dd8kg\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980811    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980826    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7318181-8bda-493f-bc63-0bf78a370280-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980838    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7318181-8bda-493f-bc63-0bf78a370280-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980850    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980862    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bd1126-9a3e-400f-8350-733958d40f2e-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980874    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779f560a-59c4-4a20-a06e-bbaf33612479-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980886    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980898    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980909    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bd1126-9a3e-400f-8350-733958d40f2e-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980923    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw5cx\" (UniqueName: \"kubernetes.io/projected/d7318181-8bda-493f-bc63-0bf78a370280-kube-api-access-qw5cx\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980937    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76xsl\" (UniqueName: \"kubernetes.io/projected/779f560a-59c4-4a20-a06e-bbaf33612479-kube-api-access-76xsl\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:02.981011 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:02.980949    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4431196d-8458-4694-a9f5-6831131c6467-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:03.205463 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.205426    2546 generic.go:356] "Generic (PLEG): container finished" podID="d7318181-8bda-493f-bc63-0bf78a370280" containerID="16df1713faa603163b9b08e792d2b1532a50de1ec92fdf74eae3ed9c0fa67821" exitCode=0
Sep 18 16:34:03.205628 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.205475    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"
Sep 18 16:34:03.205628 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.205510    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n" event={"ID":"d7318181-8bda-493f-bc63-0bf78a370280","Type":"ContainerDied","Data":"16df1713faa603163b9b08e792d2b1532a50de1ec92fdf74eae3ed9c0fa67821"}
Sep 18 16:34:03.205628 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.205551    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n" event={"ID":"d7318181-8bda-493f-bc63-0bf78a370280","Type":"ContainerDied","Data":"e6aed5a2fe04ec27053648a8fb9cd331b52bc0501e7429f9f7df90be65123bf7"}
Sep 18 16:34:03.205628 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.205568    2546 scope.go:117] "RemoveContainer" containerID="16df1713faa603163b9b08e792d2b1532a50de1ec92fdf74eae3ed9c0fa67821"
Sep 18 16:34:03.207223 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.207187    2546 generic.go:356] "Generic (PLEG): container finished" podID="4431196d-8458-4694-a9f5-6831131c6467" containerID="8eadd26e5eb2221f892091f1d620980a24fdd33f750bbc11ffabdc17562367a9" exitCode=0
Sep 18 16:34:03.207223 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.207217    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j" event={"ID":"4431196d-8458-4694-a9f5-6831131c6467","Type":"ContainerDied","Data":"8eadd26e5eb2221f892091f1d620980a24fdd33f750bbc11ffabdc17562367a9"}
Sep 18 16:34:03.207389 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.207257    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j" event={"ID":"4431196d-8458-4694-a9f5-6831131c6467","Type":"ContainerDied","Data":"8ff65739d91ac2aadcc84efccb23a3cdd1932acc3ff9791058ea62e22eb423af"}
Sep 18 16:34:03.207389 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.207223    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"
Sep 18 16:34:03.208841 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.208819    2546 generic.go:356] "Generic (PLEG): container finished" podID="779f560a-59c4-4a20-a06e-bbaf33612479" containerID="98c7362ff3e1624c991059a840d03e04ed7cd09d42f50971976bfbf397b76a81" exitCode=0
Sep 18 16:34:03.208964 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.208854    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"
Sep 18 16:34:03.209047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.208962    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5" event={"ID":"779f560a-59c4-4a20-a06e-bbaf33612479","Type":"ContainerDied","Data":"98c7362ff3e1624c991059a840d03e04ed7cd09d42f50971976bfbf397b76a81"}
Sep 18 16:34:03.209047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.208993    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5" event={"ID":"779f560a-59c4-4a20-a06e-bbaf33612479","Type":"ContainerDied","Data":"55b1d8deaabd3098360945b5acf21dfd80c9002b81f4d1a9de3640db098bdc71"}
Sep 18 16:34:03.210559 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.210535    2546 generic.go:356] "Generic (PLEG): container finished" podID="f2691e4c-7d42-4c19-8e31-b32677395c3b" containerID="e4e26a19242e67118f4fd252dc9379e33eb12347c454a6821ba929d9851cdfd3" exitCode=0
Sep 18 16:34:03.210682 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.210564    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"
Sep 18 16:34:03.210682 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.210613    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt" event={"ID":"f2691e4c-7d42-4c19-8e31-b32677395c3b","Type":"ContainerDied","Data":"e4e26a19242e67118f4fd252dc9379e33eb12347c454a6821ba929d9851cdfd3"}
Sep 18 16:34:03.210682 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.210652    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt" event={"ID":"f2691e4c-7d42-4c19-8e31-b32677395c3b","Type":"ContainerDied","Data":"74fde4a096988c46f8037bd4fbaf56430b5e7dacf758877e726ba4ba232a25ed"}
Sep 18 16:34:03.213420 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.213397    2546 generic.go:356] "Generic (PLEG): container finished" podID="97bd1126-9a3e-400f-8350-733958d40f2e" containerID="964342e43f63a572d951445f1b43a82f1928635b40830eab9760d17b0d6b4f2d" exitCode=0
Sep 18 16:34:03.213521 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.213434    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp" event={"ID":"97bd1126-9a3e-400f-8350-733958d40f2e","Type":"ContainerDied","Data":"964342e43f63a572d951445f1b43a82f1928635b40830eab9760d17b0d6b4f2d"}
Sep 18 16:34:03.213521 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.213459    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp" event={"ID":"97bd1126-9a3e-400f-8350-733958d40f2e","Type":"ContainerDied","Data":"7c91ca44c2ef41292c4f8f6eeba15fe5147ae91c13441730da152d3636557a74"}
Sep 18 16:34:03.213521 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.213436    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"
Sep 18 16:34:03.217668 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.217649    2546 scope.go:117] "RemoveContainer" containerID="16df1713faa603163b9b08e792d2b1532a50de1ec92fdf74eae3ed9c0fa67821"
Sep 18 16:34:03.218060 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:34:03.217940    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16df1713faa603163b9b08e792d2b1532a50de1ec92fdf74eae3ed9c0fa67821\": container with ID starting with 16df1713faa603163b9b08e792d2b1532a50de1ec92fdf74eae3ed9c0fa67821 not found: ID does not exist" containerID="16df1713faa603163b9b08e792d2b1532a50de1ec92fdf74eae3ed9c0fa67821"
Sep 18 16:34:03.218060 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.217980    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16df1713faa603163b9b08e792d2b1532a50de1ec92fdf74eae3ed9c0fa67821"} err="failed to get container status \"16df1713faa603163b9b08e792d2b1532a50de1ec92fdf74eae3ed9c0fa67821\": rpc error: code = NotFound desc = could not find container \"16df1713faa603163b9b08e792d2b1532a50de1ec92fdf74eae3ed9c0fa67821\": container with ID starting with 16df1713faa603163b9b08e792d2b1532a50de1ec92fdf74eae3ed9c0fa67821 not found: ID does not exist"
Sep 18 16:34:03.218060 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.218005    2546 scope.go:117] "RemoveContainer" containerID="8eadd26e5eb2221f892091f1d620980a24fdd33f750bbc11ffabdc17562367a9"
Sep 18 16:34:03.229318 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.229298    2546 scope.go:117] "RemoveContainer" containerID="8eadd26e5eb2221f892091f1d620980a24fdd33f750bbc11ffabdc17562367a9"
Sep 18 16:34:03.229586 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:34:03.229569    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eadd26e5eb2221f892091f1d620980a24fdd33f750bbc11ffabdc17562367a9\": container with ID starting with 8eadd26e5eb2221f892091f1d620980a24fdd33f750bbc11ffabdc17562367a9 not found: ID does not exist" containerID="8eadd26e5eb2221f892091f1d620980a24fdd33f750bbc11ffabdc17562367a9"
Sep 18 16:34:03.229642 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.229594    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eadd26e5eb2221f892091f1d620980a24fdd33f750bbc11ffabdc17562367a9"} err="failed to get container status \"8eadd26e5eb2221f892091f1d620980a24fdd33f750bbc11ffabdc17562367a9\": rpc error: code = NotFound desc = could not find container \"8eadd26e5eb2221f892091f1d620980a24fdd33f750bbc11ffabdc17562367a9\": container with ID starting with 8eadd26e5eb2221f892091f1d620980a24fdd33f750bbc11ffabdc17562367a9 not found: ID does not exist"
Sep 18 16:34:03.229642 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.229609    2546 scope.go:117] "RemoveContainer" containerID="98c7362ff3e1624c991059a840d03e04ed7cd09d42f50971976bfbf397b76a81"
Sep 18 16:34:03.238177 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.238161    2546 scope.go:117] "RemoveContainer" containerID="98c7362ff3e1624c991059a840d03e04ed7cd09d42f50971976bfbf397b76a81"
Sep 18 16:34:03.238434 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:34:03.238414    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c7362ff3e1624c991059a840d03e04ed7cd09d42f50971976bfbf397b76a81\": container with ID starting with 98c7362ff3e1624c991059a840d03e04ed7cd09d42f50971976bfbf397b76a81 not found: ID does not exist" containerID="98c7362ff3e1624c991059a840d03e04ed7cd09d42f50971976bfbf397b76a81"
Sep 18 16:34:03.238488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.238441    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c7362ff3e1624c991059a840d03e04ed7cd09d42f50971976bfbf397b76a81"} err="failed to get container status \"98c7362ff3e1624c991059a840d03e04ed7cd09d42f50971976bfbf397b76a81\": rpc error: code = NotFound desc = could not find container \"98c7362ff3e1624c991059a840d03e04ed7cd09d42f50971976bfbf397b76a81\": container with ID starting with 98c7362ff3e1624c991059a840d03e04ed7cd09d42f50971976bfbf397b76a81 not found: ID does not exist"
Sep 18 16:34:03.238488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.238458    2546 scope.go:117] "RemoveContainer" containerID="e4e26a19242e67118f4fd252dc9379e33eb12347c454a6821ba929d9851cdfd3"
Sep 18 16:34:03.245292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.245264    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"]
Sep 18 16:34:03.247109 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.247093    2546 scope.go:117] "RemoveContainer" containerID="e4e26a19242e67118f4fd252dc9379e33eb12347c454a6821ba929d9851cdfd3"
Sep 18 16:34:03.247385 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:34:03.247366    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e26a19242e67118f4fd252dc9379e33eb12347c454a6821ba929d9851cdfd3\": container with ID starting with e4e26a19242e67118f4fd252dc9379e33eb12347c454a6821ba929d9851cdfd3 not found: ID does not exist" containerID="e4e26a19242e67118f4fd252dc9379e33eb12347c454a6821ba929d9851cdfd3"
Sep 18 16:34:03.247455 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.247394    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e26a19242e67118f4fd252dc9379e33eb12347c454a6821ba929d9851cdfd3"} err="failed to get container status \"e4e26a19242e67118f4fd252dc9379e33eb12347c454a6821ba929d9851cdfd3\": rpc error: code = NotFound desc = could not find container \"e4e26a19242e67118f4fd252dc9379e33eb12347c454a6821ba929d9851cdfd3\": container with ID starting with e4e26a19242e67118f4fd252dc9379e33eb12347c454a6821ba929d9851cdfd3 not found: ID does not exist"
Sep 18 16:34:03.247455 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.247417    2546 scope.go:117] "RemoveContainer" containerID="964342e43f63a572d951445f1b43a82f1928635b40830eab9760d17b0d6b4f2d"
Sep 18 16:34:03.252696 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.252665    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/embedding-w0-1eec1d-kserve-5f84c9fc4f-jnnw5"]
Sep 18 16:34:03.256362 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.256342    2546 scope.go:117] "RemoveContainer" containerID="964342e43f63a572d951445f1b43a82f1928635b40830eab9760d17b0d6b4f2d"
Sep 18 16:34:03.256630 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:34:03.256615    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964342e43f63a572d951445f1b43a82f1928635b40830eab9760d17b0d6b4f2d\": container with ID starting with 964342e43f63a572d951445f1b43a82f1928635b40830eab9760d17b0d6b4f2d not found: ID does not exist" containerID="964342e43f63a572d951445f1b43a82f1928635b40830eab9760d17b0d6b4f2d"
Sep 18 16:34:03.256682 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.256635    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964342e43f63a572d951445f1b43a82f1928635b40830eab9760d17b0d6b4f2d"} err="failed to get container status \"964342e43f63a572d951445f1b43a82f1928635b40830eab9760d17b0d6b4f2d\": rpc error: code = NotFound desc = could not find container \"964342e43f63a572d951445f1b43a82f1928635b40830eab9760d17b0d6b4f2d\": container with ID starting with 964342e43f63a572d951445f1b43a82f1928635b40830eab9760d17b0d6b4f2d not found: ID does not exist"
Sep 18 16:34:03.277553 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.277524    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"]
Sep 18 16:34:03.283703 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.283679    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/premium-w0-1eec1d-kserve-685584b597-tmq5n"]
Sep 18 16:34:03.308566 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.308534    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"]
Sep 18 16:34:03.314637 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.314614    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/distinct-w0-1eec1d-kserve-77988877b-zz28j"]
Sep 18 16:34:03.336473 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.336438    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"]
Sep 18 16:34:03.341092 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.341067    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/distinct-2-w0-1eec1d-kserve-5ccf57597f-mn5vt"]
Sep 18 16:34:03.357775 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.357745    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"]
Sep 18 16:34:03.365568 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.365529    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1eec1d/simulator-w0-1eec1d-kserve-64b64d6db4-d5bpp"]
Sep 18 16:34:03.907093 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.907054    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"]
Sep 18 16:34:03.907501 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:03.907242    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy" containerID="cri-o://5cd4a6b31cc1495f61684c9b4215fbbd2af17c90cca8a287c9391bf015c73d3e" gracePeriod=30
Sep 18 16:34:04.603969 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:04.603938    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4431196d-8458-4694-a9f5-6831131c6467" path="/var/lib/kubelet/pods/4431196d-8458-4694-a9f5-6831131c6467/volumes"
Sep 18 16:34:04.604381 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:04.604366    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="779f560a-59c4-4a20-a06e-bbaf33612479" path="/var/lib/kubelet/pods/779f560a-59c4-4a20-a06e-bbaf33612479/volumes"
Sep 18 16:34:04.604721 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:04.604710    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97bd1126-9a3e-400f-8350-733958d40f2e" path="/var/lib/kubelet/pods/97bd1126-9a3e-400f-8350-733958d40f2e/volumes"
Sep 18 16:34:04.605402 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:04.605388    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7318181-8bda-493f-bc63-0bf78a370280" path="/var/lib/kubelet/pods/d7318181-8bda-493f-bc63-0bf78a370280/volumes"
Sep 18 16:34:04.605778 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:04.605766    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2691e4c-7d42-4c19-8e31-b32677395c3b" path="/var/lib/kubelet/pods/f2691e4c-7d42-4c19-8e31-b32677395c3b/volumes"
Sep 18 16:34:04.993669 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:04.993640    2546 scope.go:117] "RemoveContainer" containerID="6ccdbcb9459ef747c24f741a5340ee4314a2dfd5a3acd8c09783527e1fd978d2"
Sep 18 16:34:09.163606 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.163578    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:34:09.253755 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.253721    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-data\") pod \"055de468-2a31-4e04-b7df-4b81ca84d67f\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") "
Sep 18 16:34:09.253940 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.253766    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd2dh\" (UniqueName: \"kubernetes.io/projected/055de468-2a31-4e04-b7df-4b81ca84d67f-kube-api-access-sd2dh\") pod \"055de468-2a31-4e04-b7df-4b81ca84d67f\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") "
Sep 18 16:34:09.253940 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.253789    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055de468-2a31-4e04-b7df-4b81ca84d67f-istiod-ca-cert\") pod \"055de468-2a31-4e04-b7df-4b81ca84d67f\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") "
Sep 18 16:34:09.253940 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.253823    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-podinfo\") pod \"055de468-2a31-4e04-b7df-4b81ca84d67f\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") "
Sep 18 16:34:09.253940 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.253848    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-envoy\") pod \"055de468-2a31-4e04-b7df-4b81ca84d67f\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") "
Sep 18 16:34:09.253940 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.253887    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-workload-socket\") pod \"055de468-2a31-4e04-b7df-4b81ca84d67f\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") "
Sep 18 16:34:09.254261 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.253979    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-credential-socket\") pod \"055de468-2a31-4e04-b7df-4b81ca84d67f\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") "
Sep 18 16:34:09.254261 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.254018    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-token\") pod \"055de468-2a31-4e04-b7df-4b81ca84d67f\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") "
Sep 18 16:34:09.254261 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.254054    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-data" (OuterVolumeSpecName: "istio-data") pod "055de468-2a31-4e04-b7df-4b81ca84d67f" (UID: "055de468-2a31-4e04-b7df-4b81ca84d67f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:09.254261 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.254077    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-workload-certs\") pod \"055de468-2a31-4e04-b7df-4b81ca84d67f\" (UID: \"055de468-2a31-4e04-b7df-4b81ca84d67f\") "
Sep 18 16:34:09.254261 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.254222    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "055de468-2a31-4e04-b7df-4b81ca84d67f" (UID: "055de468-2a31-4e04-b7df-4b81ca84d67f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:09.254531 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.254253    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "055de468-2a31-4e04-b7df-4b81ca84d67f" (UID: "055de468-2a31-4e04-b7df-4b81ca84d67f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:09.254531 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.254452    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "055de468-2a31-4e04-b7df-4b81ca84d67f" (UID: "055de468-2a31-4e04-b7df-4b81ca84d67f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:09.254644 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.254535    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-data\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:09.254644 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.254550    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/055de468-2a31-4e04-b7df-4b81ca84d67f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "055de468-2a31-4e04-b7df-4b81ca84d67f" (UID: "055de468-2a31-4e04-b7df-4b81ca84d67f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:34:09.254644 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.254560    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-workload-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:09.254644 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.254577    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-credential-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:09.256361 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.256331    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "055de468-2a31-4e04-b7df-4b81ca84d67f" (UID: "055de468-2a31-4e04-b7df-4b81ca84d67f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:34:09.256479 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.256364    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "055de468-2a31-4e04-b7df-4b81ca84d67f" (UID: "055de468-2a31-4e04-b7df-4b81ca84d67f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:09.256556 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.256536    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055de468-2a31-4e04-b7df-4b81ca84d67f-kube-api-access-sd2dh" (OuterVolumeSpecName: "kube-api-access-sd2dh") pod "055de468-2a31-4e04-b7df-4b81ca84d67f" (UID: "055de468-2a31-4e04-b7df-4b81ca84d67f"). InnerVolumeSpecName "kube-api-access-sd2dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:09.256697 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.256673    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-token" (OuterVolumeSpecName: "istio-token") pod "055de468-2a31-4e04-b7df-4b81ca84d67f" (UID: "055de468-2a31-4e04-b7df-4b81ca84d67f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:09.261917 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.261894    2546 generic.go:356] "Generic (PLEG): container finished" podID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerID="5cd4a6b31cc1495f61684c9b4215fbbd2af17c90cca8a287c9391bf015c73d3e" exitCode=0
Sep 18 16:34:09.262016 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.261940    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"
Sep 18 16:34:09.262016 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.261980    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" event={"ID":"055de468-2a31-4e04-b7df-4b81ca84d67f","Type":"ContainerDied","Data":"5cd4a6b31cc1495f61684c9b4215fbbd2af17c90cca8a287c9391bf015c73d3e"}
Sep 18 16:34:09.262116 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.262061    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw" event={"ID":"055de468-2a31-4e04-b7df-4b81ca84d67f","Type":"ContainerDied","Data":"3046bf65df1e4b12bdbb70b69b7eebc41c2573ed92547d2e531ba0afd957cb85"}
Sep 18 16:34:09.262116 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.262091    2546 scope.go:117] "RemoveContainer" containerID="5cd4a6b31cc1495f61684c9b4215fbbd2af17c90cca8a287c9391bf015c73d3e"
Sep 18 16:34:09.277461 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.277445    2546 scope.go:117] "RemoveContainer" containerID="d57d4ab6f5c25ee68bf9ad66c2e4cc3777a219c4ee4a50922bc58c94985ec28a"
Sep 18 16:34:09.285720 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.285700    2546 scope.go:117] "RemoveContainer" containerID="5cd4a6b31cc1495f61684c9b4215fbbd2af17c90cca8a287c9391bf015c73d3e"
Sep 18 16:34:09.285988 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:34:09.285964    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd4a6b31cc1495f61684c9b4215fbbd2af17c90cca8a287c9391bf015c73d3e\": container with ID starting with 5cd4a6b31cc1495f61684c9b4215fbbd2af17c90cca8a287c9391bf015c73d3e not found: ID does not exist" containerID="5cd4a6b31cc1495f61684c9b4215fbbd2af17c90cca8a287c9391bf015c73d3e"
Sep 18 16:34:09.286080 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.285995    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd4a6b31cc1495f61684c9b4215fbbd2af17c90cca8a287c9391bf015c73d3e"} err="failed to get container status \"5cd4a6b31cc1495f61684c9b4215fbbd2af17c90cca8a287c9391bf015c73d3e\": rpc error: code = NotFound desc = could not find container \"5cd4a6b31cc1495f61684c9b4215fbbd2af17c90cca8a287c9391bf015c73d3e\": container with ID starting with 5cd4a6b31cc1495f61684c9b4215fbbd2af17c90cca8a287c9391bf015c73d3e not found: ID does not exist"
Sep 18 16:34:09.286080 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.286013    2546 scope.go:117] "RemoveContainer" containerID="d57d4ab6f5c25ee68bf9ad66c2e4cc3777a219c4ee4a50922bc58c94985ec28a"
Sep 18 16:34:09.286283 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:34:09.286266    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57d4ab6f5c25ee68bf9ad66c2e4cc3777a219c4ee4a50922bc58c94985ec28a\": container with ID starting with d57d4ab6f5c25ee68bf9ad66c2e4cc3777a219c4ee4a50922bc58c94985ec28a not found: ID does not exist" containerID="d57d4ab6f5c25ee68bf9ad66c2e4cc3777a219c4ee4a50922bc58c94985ec28a"
Sep 18 16:34:09.286349 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.286286    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57d4ab6f5c25ee68bf9ad66c2e4cc3777a219c4ee4a50922bc58c94985ec28a"} err="failed to get container status \"d57d4ab6f5c25ee68bf9ad66c2e4cc3777a219c4ee4a50922bc58c94985ec28a\": rpc error: code = NotFound desc = could not find container \"d57d4ab6f5c25ee68bf9ad66c2e4cc3777a219c4ee4a50922bc58c94985ec28a\": container with ID starting with d57d4ab6f5c25ee68bf9ad66c2e4cc3777a219c4ee4a50922bc58c94985ec28a not found: ID does not exist"
Sep 18 16:34:09.297220 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.297198    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"]
Sep 18 16:34:09.313449 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.313426    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1eec1d-openshift-default-f7cf48dc4-bn8lw"]
Sep 18 16:34:09.355829 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.355793    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd2dh\" (UniqueName: \"kubernetes.io/projected/055de468-2a31-4e04-b7df-4b81ca84d67f-kube-api-access-sd2dh\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:09.355829 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.355827    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/055de468-2a31-4e04-b7df-4b81ca84d67f-istiod-ca-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:09.355984 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.355836    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-podinfo\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:09.355984 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.355845    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-envoy\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:09.355984 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.355854    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/055de468-2a31-4e04-b7df-4b81ca84d67f-istio-token\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:09.355984 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:09.355861    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/055de468-2a31-4e04-b7df-4b81ca84d67f-workload-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:10.603670 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:10.603639    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" path="/var/lib/kubelet/pods/055de468-2a31-4e04-b7df-4b81ca84d67f/volumes"
Sep 18 16:34:15.555490 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:15.555451    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:34:20.638405 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638375    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq"]
Sep 18 16:34:20.638772 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638751    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2691e4c-7d42-4c19-8e31-b32677395c3b" containerName="main"
Sep 18 16:34:20.638772 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638761    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2691e4c-7d42-4c19-8e31-b32677395c3b" containerName="main"
Sep 18 16:34:20.638772 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638774    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7318181-8bda-493f-bc63-0bf78a370280" containerName="main"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638780    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7318181-8bda-493f-bc63-0bf78a370280" containerName="main"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638787    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638792    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638804    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4431196d-8458-4694-a9f5-6831131c6467" containerName="main"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638809    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4431196d-8458-4694-a9f5-6831131c6467" containerName="main"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638815    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638820    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638828    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b919690b-c250-4b19-824c-075c74e8439e" containerName="revoke-keys"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638833    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b919690b-c250-4b19-824c-075c74e8439e" containerName="revoke-keys"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638839    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bd1126-9a3e-400f-8350-733958d40f2e" containerName="main"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638843    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bd1126-9a3e-400f-8350-733958d40f2e" containerName="main"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638850    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779f560a-59c4-4a20-a06e-bbaf33612479" containerName="main"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638855    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="779f560a-59c4-4a20-a06e-bbaf33612479" containerName="main"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638861    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ea1dbef-d03f-4060-98b8-a919be1bb44b" containerName="revoke-keys"
Sep 18 16:34:20.638879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638866    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ea1dbef-d03f-4060-98b8-a919be1bb44b" containerName="revoke-keys"
Sep 18 16:34:20.639340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638918    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2691e4c-7d42-4c19-8e31-b32677395c3b" containerName="main"
Sep 18 16:34:20.639340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638925    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy"
Sep 18 16:34:20.639340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638932    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b919690b-c250-4b19-824c-075c74e8439e" containerName="revoke-keys"
Sep 18 16:34:20.639340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638938    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="055de468-2a31-4e04-b7df-4b81ca84d67f" containerName="istio-proxy"
Sep 18 16:34:20.639340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638943    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="779f560a-59c4-4a20-a06e-bbaf33612479" containerName="main"
Sep 18 16:34:20.639340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638950    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7318181-8bda-493f-bc63-0bf78a370280" containerName="main"
Sep 18 16:34:20.639340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638955    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ea1dbef-d03f-4060-98b8-a919be1bb44b" containerName="revoke-keys"
Sep 18 16:34:20.639340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638960    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4431196d-8458-4694-a9f5-6831131c6467" containerName="main"
Sep 18 16:34:20.639340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.638966    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bd1126-9a3e-400f-8350-733958d40f2e" containerName="main"
Sep 18 16:34:20.643599 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.643580    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq"
Sep 18 16:34:20.646542 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.646520    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 16:34:20.646677 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.646566    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:34:20.659483 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.659460    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq"]
Sep 18 16:34:20.767265 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.767228    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/591d6203-9055-4758-9800-45e1840f95ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq\" (UID: \"591d6203-9055-4758-9800-45e1840f95ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq"
Sep 18 16:34:20.868584 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.868551    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/591d6203-9055-4758-9800-45e1840f95ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq\" (UID: \"591d6203-9055-4758-9800-45e1840f95ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq"
Sep 18 16:34:20.869192 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.869172    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/591d6203-9055-4758-9800-45e1840f95ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq\" (UID: \"591d6203-9055-4758-9800-45e1840f95ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq"
Sep 18 16:34:20.953910 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:20.953879    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq"
Sep 18 16:34:21.241085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:21.241058    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq"]
Sep 18 16:34:21.242995 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:34:21.242970    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591d6203_9055_4758_9800_45e1840f95ec.slice/crio-f2e4f58b1fcbeb33537cdc44f25fc32b89d254d3af92a024579930ca7173eb95 WatchSource:0}: Error finding container f2e4f58b1fcbeb33537cdc44f25fc32b89d254d3af92a024579930ca7173eb95: Status 404 returned error can't find the container with id f2e4f58b1fcbeb33537cdc44f25fc32b89d254d3af92a024579930ca7173eb95
Sep 18 16:34:21.333865 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:21.333821    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq" event={"ID":"591d6203-9055-4758-9800-45e1840f95ec","Type":"ContainerStarted","Data":"f2e4f58b1fcbeb33537cdc44f25fc32b89d254d3af92a024579930ca7173eb95"}
Sep 18 16:34:22.342465 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:22.342370    2546 generic.go:356] "Generic (PLEG): container finished" podID="591d6203-9055-4758-9800-45e1840f95ec" containerID="97c8710df4169a24d9bedec7721b88847cbaf6c852af080fd57266f0049ab890" exitCode=0
Sep 18 16:34:22.342815 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:22.342457    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq" event={"ID":"591d6203-9055-4758-9800-45e1840f95ec","Type":"ContainerDied","Data":"97c8710df4169a24d9bedec7721b88847cbaf6c852af080fd57266f0049ab890"}
Sep 18 16:34:23.495691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:23.495661    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq"
Sep 18 16:34:23.595811 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:23.595779    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/591d6203-9055-4758-9800-45e1840f95ec-maas-api-service-ca\") pod \"591d6203-9055-4758-9800-45e1840f95ec\" (UID: \"591d6203-9055-4758-9800-45e1840f95ec\") "
Sep 18 16:34:23.596165 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:23.596140    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591d6203-9055-4758-9800-45e1840f95ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "591d6203-9055-4758-9800-45e1840f95ec" (UID: "591d6203-9055-4758-9800-45e1840f95ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:34:23.697208 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:23.697173    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/591d6203-9055-4758-9800-45e1840f95ec-maas-api-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:24.355701 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:24.355671    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq"
Sep 18 16:34:24.355890 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:24.355672    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq" event={"ID":"591d6203-9055-4758-9800-45e1840f95ec","Type":"ContainerDied","Data":"f2e4f58b1fcbeb33537cdc44f25fc32b89d254d3af92a024579930ca7173eb95"}
Sep 18 16:34:24.355890 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:24.355799    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2e4f58b1fcbeb33537cdc44f25fc32b89d254d3af92a024579930ca7173eb95"
Sep 18 16:34:29.196625 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:29.196573    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"]
Sep 18 16:34:31.136909 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:31.136871    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"]
Sep 18 16:34:32.534135 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:32.534095    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"]
Sep 18 16:34:33.634045 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:33.633992    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"]
Sep 18 16:34:44.215483 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.215447    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg" podUID="fc882f63-f895-454b-9e1d-51757fb7af58" containerName="main" containerID="cri-o://3ae4c01eb4172132e11975e4d23f82dc23530801a3f831e2e4935e3774dd3fc6" gracePeriod=45
Sep 18 16:34:44.461841 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.461818    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:34:44.472665 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.472610    2546 generic.go:356] "Generic (PLEG): container finished" podID="fc882f63-f895-454b-9e1d-51757fb7af58" containerID="3ae4c01eb4172132e11975e4d23f82dc23530801a3f831e2e4935e3774dd3fc6" exitCode=0
Sep 18 16:34:44.472665 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.472646    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"
Sep 18 16:34:44.472812 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.472693    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg" event={"ID":"fc882f63-f895-454b-9e1d-51757fb7af58","Type":"ContainerDied","Data":"3ae4c01eb4172132e11975e4d23f82dc23530801a3f831e2e4935e3774dd3fc6"}
Sep 18 16:34:44.472812 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.472736    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg" event={"ID":"fc882f63-f895-454b-9e1d-51757fb7af58","Type":"ContainerDied","Data":"a98e72df5672be92e6cccc67c0592a1f7990a57b7b143a424715ee839021f5e8"}
Sep 18 16:34:44.472812 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.472753    2546 scope.go:117] "RemoveContainer" containerID="3ae4c01eb4172132e11975e4d23f82dc23530801a3f831e2e4935e3774dd3fc6"
Sep 18 16:34:44.481337 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.481317    2546 scope.go:117] "RemoveContainer" containerID="3ae4c01eb4172132e11975e4d23f82dc23530801a3f831e2e4935e3774dd3fc6"
Sep 18 16:34:44.481606 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:34:44.481581    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae4c01eb4172132e11975e4d23f82dc23530801a3f831e2e4935e3774dd3fc6\": container with ID starting with 3ae4c01eb4172132e11975e4d23f82dc23530801a3f831e2e4935e3774dd3fc6 not found: ID does not exist" containerID="3ae4c01eb4172132e11975e4d23f82dc23530801a3f831e2e4935e3774dd3fc6"
Sep 18 16:34:44.481674 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.481615    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae4c01eb4172132e11975e4d23f82dc23530801a3f831e2e4935e3774dd3fc6"} err="failed to get container status \"3ae4c01eb4172132e11975e4d23f82dc23530801a3f831e2e4935e3774dd3fc6\": rpc error: code = NotFound desc = could not find container \"3ae4c01eb4172132e11975e4d23f82dc23530801a3f831e2e4935e3774dd3fc6\": container with ID starting with 3ae4c01eb4172132e11975e4d23f82dc23530801a3f831e2e4935e3774dd3fc6 not found: ID does not exist"
Sep 18 16:34:44.597592 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.597560    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppdzn\" (UniqueName: \"kubernetes.io/projected/fc882f63-f895-454b-9e1d-51757fb7af58-kube-api-access-ppdzn\") pod \"fc882f63-f895-454b-9e1d-51757fb7af58\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") "
Sep 18 16:34:44.597771 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.597602    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-home\") pod \"fc882f63-f895-454b-9e1d-51757fb7af58\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") "
Sep 18 16:34:44.597771 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.597653    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-dshm\") pod \"fc882f63-f895-454b-9e1d-51757fb7af58\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") "
Sep 18 16:34:44.597771 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.597734    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc882f63-f895-454b-9e1d-51757fb7af58-tls-certs\") pod \"fc882f63-f895-454b-9e1d-51757fb7af58\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") "
Sep 18 16:34:44.597960 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.597770    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-model-cache\") pod \"fc882f63-f895-454b-9e1d-51757fb7af58\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") "
Sep 18 16:34:44.597960 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.597872    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-tmp-dir\") pod \"fc882f63-f895-454b-9e1d-51757fb7af58\" (UID: \"fc882f63-f895-454b-9e1d-51757fb7af58\") "
Sep 18 16:34:44.597960 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.597894    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-home" (OuterVolumeSpecName: "home") pod "fc882f63-f895-454b-9e1d-51757fb7af58" (UID: "fc882f63-f895-454b-9e1d-51757fb7af58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:44.598131 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.598088    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc882f63-f895-454b-9e1d-51757fb7af58" (UID: "fc882f63-f895-454b-9e1d-51757fb7af58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:44.598295 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.598271    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:44.598295 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.598286    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc882f63-f895-454b-9e1d-51757fb7af58" (UID: "fc882f63-f895-454b-9e1d-51757fb7af58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:44.598403 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.598299    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:44.599896 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.599866    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-dshm" (OuterVolumeSpecName: "dshm") pod "fc882f63-f895-454b-9e1d-51757fb7af58" (UID: "fc882f63-f895-454b-9e1d-51757fb7af58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:44.600111 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.600079    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc882f63-f895-454b-9e1d-51757fb7af58-kube-api-access-ppdzn" (OuterVolumeSpecName: "kube-api-access-ppdzn") pod "fc882f63-f895-454b-9e1d-51757fb7af58" (UID: "fc882f63-f895-454b-9e1d-51757fb7af58"). InnerVolumeSpecName "kube-api-access-ppdzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:44.600401 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.600381    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc882f63-f895-454b-9e1d-51757fb7af58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc882f63-f895-454b-9e1d-51757fb7af58" (UID: "fc882f63-f895-454b-9e1d-51757fb7af58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:44.699946 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.699917    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:44.699946 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.699944    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc882f63-f895-454b-9e1d-51757fb7af58-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:44.699946 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.699955    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc882f63-f895-454b-9e1d-51757fb7af58-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:44.700178 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.699963    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppdzn\" (UniqueName: \"kubernetes.io/projected/fc882f63-f895-454b-9e1d-51757fb7af58-kube-api-access-ppdzn\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:44.807128 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.807057    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"]
Sep 18 16:34:44.813453 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:44.813425    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/distinct-2-w4-4f2d8a-kserve-6b8876d9cf-62dwg"]
Sep 18 16:34:46.156996 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.156950    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l" podUID="a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a" containerName="main" containerID="cri-o://3ccdf82b844dd9e84eb61620fe618f81dea86a5ec38d1f935724ede3bb7baadd" gracePeriod=45
Sep 18 16:34:46.407494 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.407435    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:34:46.489170 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.489131    2546 generic.go:356] "Generic (PLEG): container finished" podID="a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a" containerID="3ccdf82b844dd9e84eb61620fe618f81dea86a5ec38d1f935724ede3bb7baadd" exitCode=0
Sep 18 16:34:46.489366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.489186    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"
Sep 18 16:34:46.489366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.489219    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l" event={"ID":"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a","Type":"ContainerDied","Data":"3ccdf82b844dd9e84eb61620fe618f81dea86a5ec38d1f935724ede3bb7baadd"}
Sep 18 16:34:46.489366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.489271    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l" event={"ID":"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a","Type":"ContainerDied","Data":"963c9c5ac88560b8f46c9088a3b727439e92f49ff518a8eb8b93e4579b73918c"}
Sep 18 16:34:46.489366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.489297    2546 scope.go:117] "RemoveContainer" containerID="3ccdf82b844dd9e84eb61620fe618f81dea86a5ec38d1f935724ede3bb7baadd"
Sep 18 16:34:46.498277 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.498255    2546 scope.go:117] "RemoveContainer" containerID="3ccdf82b844dd9e84eb61620fe618f81dea86a5ec38d1f935724ede3bb7baadd"
Sep 18 16:34:46.498561 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:34:46.498541    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ccdf82b844dd9e84eb61620fe618f81dea86a5ec38d1f935724ede3bb7baadd\": container with ID starting with 3ccdf82b844dd9e84eb61620fe618f81dea86a5ec38d1f935724ede3bb7baadd not found: ID does not exist" containerID="3ccdf82b844dd9e84eb61620fe618f81dea86a5ec38d1f935724ede3bb7baadd"
Sep 18 16:34:46.498635 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.498570    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ccdf82b844dd9e84eb61620fe618f81dea86a5ec38d1f935724ede3bb7baadd"} err="failed to get container status \"3ccdf82b844dd9e84eb61620fe618f81dea86a5ec38d1f935724ede3bb7baadd\": rpc error: code = NotFound desc = could not find container \"3ccdf82b844dd9e84eb61620fe618f81dea86a5ec38d1f935724ede3bb7baadd\": container with ID starting with 3ccdf82b844dd9e84eb61620fe618f81dea86a5ec38d1f935724ede3bb7baadd not found: ID does not exist"
Sep 18 16:34:46.515915 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.515889    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-home\") pod \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") "
Sep 18 16:34:46.516054 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.515962    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-model-cache\") pod \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") "
Sep 18 16:34:46.516104 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.516060    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-tmp-dir\") pod \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") "
Sep 18 16:34:46.516104 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.516094    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-tls-certs\") pod \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") "
Sep 18 16:34:46.516199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.516116    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f6x4\" (UniqueName: \"kubernetes.io/projected/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-kube-api-access-9f6x4\") pod \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") "
Sep 18 16:34:46.516199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.516151    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-dshm\") pod \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\" (UID: \"a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a\") "
Sep 18 16:34:46.516199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.516161    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-home" (OuterVolumeSpecName: "home") pod "a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a" (UID: "a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:46.516199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.516191    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a" (UID: "a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:46.516414 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.516346    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a" (UID: "a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:46.516488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.516473    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:46.516557 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.516490    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:46.516557 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.516499    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:46.518271 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.518237    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-kube-api-access-9f6x4" (OuterVolumeSpecName: "kube-api-access-9f6x4") pod "a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a" (UID: "a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a"). InnerVolumeSpecName "kube-api-access-9f6x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:46.518356 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.518274    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a" (UID: "a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:46.518397 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.518352    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-dshm" (OuterVolumeSpecName: "dshm") pod "a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a" (UID: "a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:46.603773 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.603740    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc882f63-f895-454b-9e1d-51757fb7af58" path="/var/lib/kubelet/pods/fc882f63-f895-454b-9e1d-51757fb7af58/volumes"
Sep 18 16:34:46.617397 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.617375    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:46.617397 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.617397    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f6x4\" (UniqueName: \"kubernetes.io/projected/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-kube-api-access-9f6x4\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:46.617521 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.617408    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:46.822088 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.822054    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"]
Sep 18 16:34:46.835779 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:46.835742    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/distinct-w4-4f2d8a-kserve-5bc5fcdd4d-g749l"]
Sep 18 16:34:47.553020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.552974    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft" podUID="a8cca791-527d-4558-86b8-a0833983da93" containerName="main" containerID="cri-o://7dddcaafbdd2195e539f9ba3cfa51606281dd06d71a98e4933ae314fcc9f06ff" gracePeriod=45
Sep 18 16:34:47.813350 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.813293    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:34:47.932220 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.932190    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-tmp-dir\") pod \"a8cca791-527d-4558-86b8-a0833983da93\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") "
Sep 18 16:34:47.932388 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.932271    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cca791-527d-4558-86b8-a0833983da93-tls-certs\") pod \"a8cca791-527d-4558-86b8-a0833983da93\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") "
Sep 18 16:34:47.932388 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.932326    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gmfh\" (UniqueName: \"kubernetes.io/projected/a8cca791-527d-4558-86b8-a0833983da93-kube-api-access-6gmfh\") pod \"a8cca791-527d-4558-86b8-a0833983da93\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") "
Sep 18 16:34:47.932388 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.932384    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-home\") pod \"a8cca791-527d-4558-86b8-a0833983da93\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") "
Sep 18 16:34:47.932588 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.932442    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-model-cache\") pod \"a8cca791-527d-4558-86b8-a0833983da93\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") "
Sep 18 16:34:47.932588 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.932479    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8cca791-527d-4558-86b8-a0833983da93" (UID: "a8cca791-527d-4558-86b8-a0833983da93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:47.932588 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.932492    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-dshm\") pod \"a8cca791-527d-4558-86b8-a0833983da93\" (UID: \"a8cca791-527d-4558-86b8-a0833983da93\") "
Sep 18 16:34:47.932760 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.932718    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8cca791-527d-4558-86b8-a0833983da93" (UID: "a8cca791-527d-4558-86b8-a0833983da93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:47.932829 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.932732    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-home" (OuterVolumeSpecName: "home") pod "a8cca791-527d-4558-86b8-a0833983da93" (UID: "a8cca791-527d-4558-86b8-a0833983da93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:47.933071 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.933046    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:47.933147 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.933077    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:47.933147 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.933091    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:47.934404 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.934379    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cca791-527d-4558-86b8-a0833983da93-kube-api-access-6gmfh" (OuterVolumeSpecName: "kube-api-access-6gmfh") pod "a8cca791-527d-4558-86b8-a0833983da93" (UID: "a8cca791-527d-4558-86b8-a0833983da93"). InnerVolumeSpecName "kube-api-access-6gmfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:47.934512 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.934416    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-dshm" (OuterVolumeSpecName: "dshm") pod "a8cca791-527d-4558-86b8-a0833983da93" (UID: "a8cca791-527d-4558-86b8-a0833983da93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:47.934512 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:47.934427    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cca791-527d-4558-86b8-a0833983da93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8cca791-527d-4558-86b8-a0833983da93" (UID: "a8cca791-527d-4558-86b8-a0833983da93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:48.034059 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.034012    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8cca791-527d-4558-86b8-a0833983da93-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:48.034059 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.034055    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8cca791-527d-4558-86b8-a0833983da93-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:48.034231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.034067    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gmfh\" (UniqueName: \"kubernetes.io/projected/a8cca791-527d-4558-86b8-a0833983da93-kube-api-access-6gmfh\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:48.504941 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.504910    2546 generic.go:356] "Generic (PLEG): container finished" podID="a8cca791-527d-4558-86b8-a0833983da93" containerID="7dddcaafbdd2195e539f9ba3cfa51606281dd06d71a98e4933ae314fcc9f06ff" exitCode=0
Sep 18 16:34:48.505153 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.504968    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"
Sep 18 16:34:48.505153 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.504997    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft" event={"ID":"a8cca791-527d-4558-86b8-a0833983da93","Type":"ContainerDied","Data":"7dddcaafbdd2195e539f9ba3cfa51606281dd06d71a98e4933ae314fcc9f06ff"}
Sep 18 16:34:48.505153 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.505059    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft" event={"ID":"a8cca791-527d-4558-86b8-a0833983da93","Type":"ContainerDied","Data":"c9700932e2559cf9aed9ec3274f6ed12748b04832479e2ac18c8248f6c2ba21a"}
Sep 18 16:34:48.505153 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.505085    2546 scope.go:117] "RemoveContainer" containerID="7dddcaafbdd2195e539f9ba3cfa51606281dd06d71a98e4933ae314fcc9f06ff"
Sep 18 16:34:48.514902 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.514885    2546 scope.go:117] "RemoveContainer" containerID="7dddcaafbdd2195e539f9ba3cfa51606281dd06d71a98e4933ae314fcc9f06ff"
Sep 18 16:34:48.515175 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:34:48.515156    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dddcaafbdd2195e539f9ba3cfa51606281dd06d71a98e4933ae314fcc9f06ff\": container with ID starting with 7dddcaafbdd2195e539f9ba3cfa51606281dd06d71a98e4933ae314fcc9f06ff not found: ID does not exist" containerID="7dddcaafbdd2195e539f9ba3cfa51606281dd06d71a98e4933ae314fcc9f06ff"
Sep 18 16:34:48.515237 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.515182    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dddcaafbdd2195e539f9ba3cfa51606281dd06d71a98e4933ae314fcc9f06ff"} err="failed to get container status \"7dddcaafbdd2195e539f9ba3cfa51606281dd06d71a98e4933ae314fcc9f06ff\": rpc error: code = NotFound desc = could not find container \"7dddcaafbdd2195e539f9ba3cfa51606281dd06d71a98e4933ae314fcc9f06ff\": container with ID starting with 7dddcaafbdd2195e539f9ba3cfa51606281dd06d71a98e4933ae314fcc9f06ff not found: ID does not exist"
Sep 18 16:34:48.538888 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.538865    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"]
Sep 18 16:34:48.546090 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.546067    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/premium-w4-4f2d8a-kserve-6558b99857-sj6ft"]
Sep 18 16:34:48.602638 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.602618    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a" path="/var/lib/kubelet/pods/a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a/volumes"
Sep 18 16:34:48.603009 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.602995    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8cca791-527d-4558-86b8-a0833983da93" path="/var/lib/kubelet/pods/a8cca791-527d-4558-86b8-a0833983da93/volumes"
Sep 18 16:34:48.652976 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.652949    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc" podUID="696e445d-bd03-4d5e-b689-76d70ff3917b" containerName="main" containerID="cri-o://9e77605415e05f6821bf8baf529dfbb73c2e82c5e27004d8b129fe1ad9c0a311" gracePeriod=45
Sep 18 16:34:48.931832 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:48.931806    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:34:49.044364 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.044289    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-dshm\") pod \"696e445d-bd03-4d5e-b689-76d70ff3917b\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") "
Sep 18 16:34:49.044364 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.044361    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-home\") pod \"696e445d-bd03-4d5e-b689-76d70ff3917b\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") "
Sep 18 16:34:49.044571 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.044387    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-model-cache\") pod \"696e445d-bd03-4d5e-b689-76d70ff3917b\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") "
Sep 18 16:34:49.044571 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.044427    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-tmp-dir\") pod \"696e445d-bd03-4d5e-b689-76d70ff3917b\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") "
Sep 18 16:34:49.044571 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.044451    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696e445d-bd03-4d5e-b689-76d70ff3917b-tls-certs\") pod \"696e445d-bd03-4d5e-b689-76d70ff3917b\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") "
Sep 18 16:34:49.044571 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.044514    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jnx5\" (UniqueName: \"kubernetes.io/projected/696e445d-bd03-4d5e-b689-76d70ff3917b-kube-api-access-5jnx5\") pod \"696e445d-bd03-4d5e-b689-76d70ff3917b\" (UID: \"696e445d-bd03-4d5e-b689-76d70ff3917b\") "
Sep 18 16:34:49.044777 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.044637    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "696e445d-bd03-4d5e-b689-76d70ff3917b" (UID: "696e445d-bd03-4d5e-b689-76d70ff3917b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:49.044777 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.044669    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-home" (OuterVolumeSpecName: "home") pod "696e445d-bd03-4d5e-b689-76d70ff3917b" (UID: "696e445d-bd03-4d5e-b689-76d70ff3917b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:49.044777 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.044685    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-model-cache" (OuterVolumeSpecName: "model-cache") pod "696e445d-bd03-4d5e-b689-76d70ff3917b" (UID: "696e445d-bd03-4d5e-b689-76d70ff3917b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:49.044912 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.044894    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:49.044961 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.044917    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:49.044961 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.044926    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:49.046422 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.046393    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/696e445d-bd03-4d5e-b689-76d70ff3917b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "696e445d-bd03-4d5e-b689-76d70ff3917b" (UID: "696e445d-bd03-4d5e-b689-76d70ff3917b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:49.046537 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.046460    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-dshm" (OuterVolumeSpecName: "dshm") pod "696e445d-bd03-4d5e-b689-76d70ff3917b" (UID: "696e445d-bd03-4d5e-b689-76d70ff3917b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:49.046537 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.046499    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696e445d-bd03-4d5e-b689-76d70ff3917b-kube-api-access-5jnx5" (OuterVolumeSpecName: "kube-api-access-5jnx5") pod "696e445d-bd03-4d5e-b689-76d70ff3917b" (UID: "696e445d-bd03-4d5e-b689-76d70ff3917b"). InnerVolumeSpecName "kube-api-access-5jnx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:49.146206 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.146176    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/696e445d-bd03-4d5e-b689-76d70ff3917b-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:49.146206 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.146201    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jnx5\" (UniqueName: \"kubernetes.io/projected/696e445d-bd03-4d5e-b689-76d70ff3917b-kube-api-access-5jnx5\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:49.146206 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.146211    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/696e445d-bd03-4d5e-b689-76d70ff3917b-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:34:49.512337 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.512303    2546 generic.go:356] "Generic (PLEG): container finished" podID="696e445d-bd03-4d5e-b689-76d70ff3917b" containerID="9e77605415e05f6821bf8baf529dfbb73c2e82c5e27004d8b129fe1ad9c0a311" exitCode=0
Sep 18 16:34:49.512510 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.512347    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"
Sep 18 16:34:49.512510 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.512395    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc" event={"ID":"696e445d-bd03-4d5e-b689-76d70ff3917b","Type":"ContainerDied","Data":"9e77605415e05f6821bf8baf529dfbb73c2e82c5e27004d8b129fe1ad9c0a311"}
Sep 18 16:34:49.512510 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.512439    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc" event={"ID":"696e445d-bd03-4d5e-b689-76d70ff3917b","Type":"ContainerDied","Data":"3dca1406ec4739d1a3dc6aac88e7474de93a1e9064089460180072f8640fdaf1"}
Sep 18 16:34:49.512510 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.512464    2546 scope.go:117] "RemoveContainer" containerID="9e77605415e05f6821bf8baf529dfbb73c2e82c5e27004d8b129fe1ad9c0a311"
Sep 18 16:34:49.521535 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.521519    2546 scope.go:117] "RemoveContainer" containerID="9e77605415e05f6821bf8baf529dfbb73c2e82c5e27004d8b129fe1ad9c0a311"
Sep 18 16:34:49.521771 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:34:49.521754    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e77605415e05f6821bf8baf529dfbb73c2e82c5e27004d8b129fe1ad9c0a311\": container with ID starting with 9e77605415e05f6821bf8baf529dfbb73c2e82c5e27004d8b129fe1ad9c0a311 not found: ID does not exist" containerID="9e77605415e05f6821bf8baf529dfbb73c2e82c5e27004d8b129fe1ad9c0a311"
Sep 18 16:34:49.521841 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.521775    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e77605415e05f6821bf8baf529dfbb73c2e82c5e27004d8b129fe1ad9c0a311"} err="failed to get container status \"9e77605415e05f6821bf8baf529dfbb73c2e82c5e27004d8b129fe1ad9c0a311\": rpc error: code = NotFound desc = could not find container \"9e77605415e05f6821bf8baf529dfbb73c2e82c5e27004d8b129fe1ad9c0a311\": container with ID starting with 9e77605415e05f6821bf8baf529dfbb73c2e82c5e27004d8b129fe1ad9c0a311 not found: ID does not exist"
Sep 18 16:34:49.545054 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.545018    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"]
Sep 18 16:34:49.552166 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:49.552145    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4f2d8a/simulator-w4-4f2d8a-kserve-667dbcffb4-5w6hc"]
Sep 18 16:34:50.278699 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:50.278657    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf"]
Sep 18 16:34:50.283717 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:50.283687    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4f2d8a-29829150-mmfxf"]
Sep 18 16:34:50.603118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:50.603042    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="696e445d-bd03-4d5e-b689-76d70ff3917b" path="/var/lib/kubelet/pods/696e445d-bd03-4d5e-b689-76d70ff3917b/volumes"
Sep 18 16:34:50.603354 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:50.603341    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2360b9a-50ee-488e-b34d-52a03308b0bc" path="/var/lib/kubelet/pods/d2360b9a-50ee-488e-b34d-52a03308b0bc/volumes"
Sep 18 16:34:50.928707 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:50.928624    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq"]
Sep 18 16:34:50.943926 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:50.943898    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4f2d8a-924pq"]
Sep 18 16:34:52.603764 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:52.603731    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="591d6203-9055-4758-9800-45e1840f95ec" path="/var/lib/kubelet/pods/591d6203-9055-4758-9800-45e1840f95ec/volumes"
Sep 18 16:34:59.709787 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:59.709710    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:34:59.721921 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:59.721887    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"]
Sep 18 16:34:59.722171 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:34:59.722147    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerName="istio-proxy" containerID="cri-o://88b1721921d924bab215bbe10c07584fbe0593ebd16265a1f1f7c9dc862e1db2" gracePeriod=30
Sep 18 16:35:04.990743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:04.990676    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:35:05.047681 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.047656    2546 scope.go:117] "RemoveContainer" containerID="81ddef6053096eeb1befd1ed11541a8862d598e1e1118951900b8a7ded514e58"
Sep 18 16:35:05.056047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.056007    2546 scope.go:117] "RemoveContainer" containerID="88b1721921d924bab215bbe10c07584fbe0593ebd16265a1f1f7c9dc862e1db2"
Sep 18 16:35:05.096609 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.096589    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-envoy\") pod \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") "
Sep 18 16:35:05.096698 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.096654    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-data\") pod \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") "
Sep 18 16:35:05.096757 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.096695    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-podinfo\") pod \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") "
Sep 18 16:35:05.096757 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.096725    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-workload-socket\") pod \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") "
Sep 18 16:35:05.096875 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.096808    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk4cm\" (UniqueName: \"kubernetes.io/projected/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-kube-api-access-hk4cm\") pod \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") "
Sep 18 16:35:05.096875 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.096843    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-credential-socket\") pod \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") "
Sep 18 16:35:05.096981 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.096913    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-workload-certs\") pod \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") "
Sep 18 16:35:05.096981 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.096950    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-token\") pod \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") "
Sep 18 16:35:05.096981 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.096946    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9ca1259-e61b-4a6d-81fd-bd67c87d0517" (UID: "d9ca1259-e61b-4a6d-81fd-bd67c87d0517"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:05.096981 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.096967    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istiod-ca-cert\") pod \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\" (UID: \"d9ca1259-e61b-4a6d-81fd-bd67c87d0517\") "
Sep 18 16:35:05.097214 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.097107    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9ca1259-e61b-4a6d-81fd-bd67c87d0517" (UID: "d9ca1259-e61b-4a6d-81fd-bd67c87d0517"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:05.097275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.097232    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9ca1259-e61b-4a6d-81fd-bd67c87d0517" (UID: "d9ca1259-e61b-4a6d-81fd-bd67c87d0517"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:05.097399 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.097382    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-workload-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:35:05.097467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.097397    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9ca1259-e61b-4a6d-81fd-bd67c87d0517" (UID: "d9ca1259-e61b-4a6d-81fd-bd67c87d0517"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:35:05.097467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.097401    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9ca1259-e61b-4a6d-81fd-bd67c87d0517" (UID: "d9ca1259-e61b-4a6d-81fd-bd67c87d0517"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:05.097467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.097405    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-workload-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:35:05.097467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.097448    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-data\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:35:05.098961 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.098930    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9ca1259-e61b-4a6d-81fd-bd67c87d0517" (UID: "d9ca1259-e61b-4a6d-81fd-bd67c87d0517"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:35:05.099090 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.099071    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9ca1259-e61b-4a6d-81fd-bd67c87d0517" (UID: "d9ca1259-e61b-4a6d-81fd-bd67c87d0517"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:35:05.099157 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.099137    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9ca1259-e61b-4a6d-81fd-bd67c87d0517" (UID: "d9ca1259-e61b-4a6d-81fd-bd67c87d0517"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:35:05.099232 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.099213    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-kube-api-access-hk4cm" (OuterVolumeSpecName: "kube-api-access-hk4cm") pod "d9ca1259-e61b-4a6d-81fd-bd67c87d0517" (UID: "d9ca1259-e61b-4a6d-81fd-bd67c87d0517"). InnerVolumeSpecName "kube-api-access-hk4cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:35:05.198953 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.198910    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-podinfo\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:35:05.198953 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.198942    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk4cm\" (UniqueName: \"kubernetes.io/projected/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-kube-api-access-hk4cm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:35:05.198953 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.198954    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-credential-socket\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:35:05.198953 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.198963    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-token\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:35:05.199243 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.198971    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istiod-ca-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:35:05.199243 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.198979    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9ca1259-e61b-4a6d-81fd-bd67c87d0517-istio-envoy\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:35:05.606438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.606406    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" event={"ID":"d9ca1259-e61b-4a6d-81fd-bd67c87d0517","Type":"ContainerDied","Data":"88b1721921d924bab215bbe10c07584fbe0593ebd16265a1f1f7c9dc862e1db2"}
Sep 18 16:35:05.606637 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.606459    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k" event={"ID":"d9ca1259-e61b-4a6d-81fd-bd67c87d0517","Type":"ContainerDied","Data":"3ed9de66281246dd1b20247ed3f239d62b6300bfcfcd095d4c53136328c4a8d3"}
Sep 18 16:35:05.606637 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.606419    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"
Sep 18 16:35:05.668387 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.668349    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"]
Sep 18 16:35:05.673885 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:05.673856    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4f2d8a-openshift-default-69bf5668c4-dzj8k"]
Sep 18 16:35:06.603600 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:06.603567    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" path="/var/lib/kubelet/pods/d9ca1259-e61b-4a6d-81fd-bd67c87d0517/volumes"
Sep 18 16:35:16.068363 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:16.068324    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:35:30.067645 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:30.067604    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:35:37.397871 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.397831    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7582-b1db70"]
Sep 18 16:35:37.398292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398249    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc882f63-f895-454b-9e1d-51757fb7af58" containerName="main"
Sep 18 16:35:37.398292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398261    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc882f63-f895-454b-9e1d-51757fb7af58" containerName="main"
Sep 18 16:35:37.398292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398276    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8cca791-527d-4558-86b8-a0833983da93" containerName="main"
Sep 18 16:35:37.398292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398282    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8cca791-527d-4558-86b8-a0833983da93" containerName="main"
Sep 18 16:35:37.398292 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398292    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerName="istio-proxy"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398299    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerName="istio-proxy"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398304    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a" containerName="main"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398309    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a" containerName="main"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398314    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591d6203-9055-4758-9800-45e1840f95ec" containerName="revoke-keys"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398319    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="591d6203-9055-4758-9800-45e1840f95ec" containerName="revoke-keys"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398329    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="696e445d-bd03-4d5e-b689-76d70ff3917b" containerName="main"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398335    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="696e445d-bd03-4d5e-b689-76d70ff3917b" containerName="main"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398346    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerName="istio-proxy"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398352    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerName="istio-proxy"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398407    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8cca791-527d-4558-86b8-a0833983da93" containerName="main"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398415    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0e395b1-2d3d-407e-b2e6-7e97b3ecd85a" containerName="main"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398420    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc882f63-f895-454b-9e1d-51757fb7af58" containerName="main"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398427    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="591d6203-9055-4758-9800-45e1840f95ec" containerName="revoke-keys"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398434    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerName="istio-proxy"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398440    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="696e445d-bd03-4d5e-b689-76d70ff3917b" containerName="main"
Sep 18 16:35:37.398457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.398445    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ca1259-e61b-4a6d-81fd-bd67c87d0517" containerName="istio-proxy"
Sep 18 16:35:37.401523 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.401505    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7582-b1db70"
Sep 18 16:35:37.405206 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.405183    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6q5gq\""
Sep 18 16:35:37.413401 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.413376    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7582-b1db70"]
Sep 18 16:35:37.504557 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.504515    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqnpn\" (UniqueName: \"kubernetes.io/projected/e5591ce0-9e0a-46c2-8241-583d499b4bff-kube-api-access-tqnpn\") pod \"test-curl-7582-b1db70\" (UID: \"e5591ce0-9e0a-46c2-8241-583d499b4bff\") " pod="opendatahub/test-curl-7582-b1db70"
Sep 18 16:35:37.605183 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.605144    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqnpn\" (UniqueName: \"kubernetes.io/projected/e5591ce0-9e0a-46c2-8241-583d499b4bff-kube-api-access-tqnpn\") pod \"test-curl-7582-b1db70\" (UID: \"e5591ce0-9e0a-46c2-8241-583d499b4bff\") " pod="opendatahub/test-curl-7582-b1db70"
Sep 18 16:35:37.621834 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.621803    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqnpn\" (UniqueName: \"kubernetes.io/projected/e5591ce0-9e0a-46c2-8241-583d499b4bff-kube-api-access-tqnpn\") pod \"test-curl-7582-b1db70\" (UID: \"e5591ce0-9e0a-46c2-8241-583d499b4bff\") " pod="opendatahub/test-curl-7582-b1db70"
Sep 18 16:35:37.711153 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.711115    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7582-b1db70"
Sep 18 16:35:37.997969 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:37.997932    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7582-b1db70"]
Sep 18 16:35:37.999950 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:35:37.999923    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5591ce0_9e0a_46c2_8241_583d499b4bff.slice/crio-06c387c74c047af31f22c46675c4fd997e78b9e0a1bc41b535d48014a63fd04e WatchSource:0}: Error finding container 06c387c74c047af31f22c46675c4fd997e78b9e0a1bc41b535d48014a63fd04e: Status 404 returned error can't find the container with id 06c387c74c047af31f22c46675c4fd997e78b9e0a1bc41b535d48014a63fd04e
Sep 18 16:35:38.805756 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:38.805710    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7582-b1db70" event={"ID":"e5591ce0-9e0a-46c2-8241-583d499b4bff","Type":"ContainerStarted","Data":"b6981bc519e9170037d365be5e5f9d4a4f284b5e53f864f9043a49022156cfff"}
Sep 18 16:35:38.805756 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:38.805762    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7582-b1db70" event={"ID":"e5591ce0-9e0a-46c2-8241-583d499b4bff","Type":"ContainerStarted","Data":"06c387c74c047af31f22c46675c4fd997e78b9e0a1bc41b535d48014a63fd04e"}
Sep 18 16:35:38.867046 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:38.866967    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7582-b1db70" podStartSLOduration=1.866946483 podStartE2EDuration="1.866946483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:35:38.863788795 +0000 UTC m=+1834.772569148" watchObservedRunningTime="2026-09-18 16:35:38.866946483 +0000 UTC m=+1834.775726836"
Sep 18 16:35:39.662063 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:39.661957    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7582-b1db70"]
Sep 18 16:35:39.667106 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:39.667080    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7582-b1db70"]
Sep 18 16:35:39.811276 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:35:39.811241    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7582-b1db70" container="test-curl-7582-b1db70"
Sep 18 16:35:40.816397 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:40.816360    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7582-b1db70" podUID="e5591ce0-9e0a-46c2-8241-583d499b4bff" containerName="test-curl-7582-b1db70" containerID="cri-o://b6981bc519e9170037d365be5e5f9d4a4f284b5e53f864f9043a49022156cfff" gracePeriod=2
Sep 18 16:35:43.009149 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:43.009122    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7582-b1db70"
Sep 18 16:35:43.160992 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:43.160903    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqnpn\" (UniqueName: \"kubernetes.io/projected/e5591ce0-9e0a-46c2-8241-583d499b4bff-kube-api-access-tqnpn\") pod \"e5591ce0-9e0a-46c2-8241-583d499b4bff\" (UID: \"e5591ce0-9e0a-46c2-8241-583d499b4bff\") "
Sep 18 16:35:43.163149 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:43.163125    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5591ce0-9e0a-46c2-8241-583d499b4bff-kube-api-access-tqnpn" (OuterVolumeSpecName: "kube-api-access-tqnpn") pod "e5591ce0-9e0a-46c2-8241-583d499b4bff" (UID: "e5591ce0-9e0a-46c2-8241-583d499b4bff"). InnerVolumeSpecName "kube-api-access-tqnpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:35:43.262578 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:43.262540    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqnpn\" (UniqueName: \"kubernetes.io/projected/e5591ce0-9e0a-46c2-8241-583d499b4bff-kube-api-access-tqnpn\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:35:43.836017 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:43.835978    2546 generic.go:356] "Generic (PLEG): container finished" podID="e5591ce0-9e0a-46c2-8241-583d499b4bff" containerID="b6981bc519e9170037d365be5e5f9d4a4f284b5e53f864f9043a49022156cfff" exitCode=137
Sep 18 16:35:43.836258 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:43.836053    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7582-b1db70"
Sep 18 16:35:43.836258 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:43.836072    2546 scope.go:117] "RemoveContainer" containerID="b6981bc519e9170037d365be5e5f9d4a4f284b5e53f864f9043a49022156cfff"
Sep 18 16:35:43.846324 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:43.846301    2546 scope.go:117] "RemoveContainer" containerID="b6981bc519e9170037d365be5e5f9d4a4f284b5e53f864f9043a49022156cfff"
Sep 18 16:35:43.846614 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:35:43.846595    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6981bc519e9170037d365be5e5f9d4a4f284b5e53f864f9043a49022156cfff\": container with ID starting with b6981bc519e9170037d365be5e5f9d4a4f284b5e53f864f9043a49022156cfff not found: ID does not exist" containerID="b6981bc519e9170037d365be5e5f9d4a4f284b5e53f864f9043a49022156cfff"
Sep 18 16:35:43.846685 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:43.846628    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6981bc519e9170037d365be5e5f9d4a4f284b5e53f864f9043a49022156cfff"} err="failed to get container status \"b6981bc519e9170037d365be5e5f9d4a4f284b5e53f864f9043a49022156cfff\": rpc error: code = NotFound desc = could not find container \"b6981bc519e9170037d365be5e5f9d4a4f284b5e53f864f9043a49022156cfff\": container with ID starting with b6981bc519e9170037d365be5e5f9d4a4f284b5e53f864f9043a49022156cfff not found: ID does not exist"
Sep 18 16:35:43.849116 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:35:43.849091    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7582-b1db70\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="e5591ce0-9e0a-46c2-8241-583d499b4bff" pod="opendatahub/test-curl-7582-b1db70"
Sep 18 16:35:44.601652 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:35:44.601616    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7582-b1db70\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="e5591ce0-9e0a-46c2-8241-583d499b4bff" pod="opendatahub/test-curl-7582-b1db70"
Sep 18 16:35:44.603449 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:44.603414    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5591ce0-9e0a-46c2-8241-583d499b4bff" path="/var/lib/kubelet/pods/e5591ce0-9e0a-46c2-8241-583d499b4bff/volumes"
Sep 18 16:35:44.903286 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:44.903201    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:35:50.173405 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.173355    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"]
Sep 18 16:35:50.173793 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.173771    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5591ce0-9e0a-46c2-8241-583d499b4bff" containerName="test-curl-7582-b1db70"
Sep 18 16:35:50.173793 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.173784    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5591ce0-9e0a-46c2-8241-583d499b4bff" containerName="test-curl-7582-b1db70"
Sep 18 16:35:50.173864 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.173859    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5591ce0-9e0a-46c2-8241-583d499b4bff" containerName="test-curl-7582-b1db70"
Sep 18 16:35:50.178339 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.178316    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.181395 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.181359    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"sub-test-model-a75117-kserve-self-signed-certs\""
Sep 18 16:35:50.182359 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.182336    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"kube-root-ca.crt\""
Sep 18 16:35:50.182483 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.182360    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"openshift-service-ca.crt\""
Sep 18 16:35:50.182483 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.182443    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"default-dockercfg-jshkt\""
Sep 18 16:35:50.205299 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.205266    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"]
Sep 18 16:35:50.234777 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.234741    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l49s5\" (UniqueName: \"kubernetes.io/projected/de443cc3-cf0a-452c-969f-a404401d7e8f-kube-api-access-l49s5\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.234968 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.234942    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-model-cache\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.235117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.235097    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de443cc3-cf0a-452c-969f-a404401d7e8f-tls-certs\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.235269 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.235250    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-home\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.235350 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.235306    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-tmp-dir\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.235414 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.235395    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-dshm\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.335989 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.335953    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de443cc3-cf0a-452c-969f-a404401d7e8f-tls-certs\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.336204 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.336014    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-home\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.336204 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.336084    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-tmp-dir\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.336204 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.336105    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-dshm\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.336204 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.336146    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l49s5\" (UniqueName: \"kubernetes.io/projected/de443cc3-cf0a-452c-969f-a404401d7e8f-kube-api-access-l49s5\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.336204 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.336179    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-model-cache\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.336597 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.336570    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-home\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.336675 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.336597    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-tmp-dir\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.336675 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.336631    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-model-cache\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.338364 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.338340    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-dshm\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.338597 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.338564    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de443cc3-cf0a-452c-969f-a404401d7e8f-tls-certs\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.349889 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.349866    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l49s5\" (UniqueName: \"kubernetes.io/projected/de443cc3-cf0a-452c-969f-a404401d7e8f-kube-api-access-l49s5\") pod \"sub-test-model-a75117-kserve-76c4d85cf5-wfprt\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") " pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.489111 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.489063    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:35:50.781661 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.781632    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"]
Sep 18 16:35:50.783717 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:35:50.783689    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde443cc3_cf0a_452c_969f_a404401d7e8f.slice/crio-65121af6ff1886f7b4cb6b7e783dd45d2e195d5c80b0a9201100a993a79c9e24 WatchSource:0}: Error finding container 65121af6ff1886f7b4cb6b7e783dd45d2e195d5c80b0a9201100a993a79c9e24: Status 404 returned error can't find the container with id 65121af6ff1886f7b4cb6b7e783dd45d2e195d5c80b0a9201100a993a79c9e24
Sep 18 16:35:50.880939 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.880901    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt" event={"ID":"de443cc3-cf0a-452c-969f-a404401d7e8f","Type":"ContainerStarted","Data":"02c7e72e4a55b444d56f237b35c7cb7f012983159be03f5fe3493e8e5fb63cfb"}
Sep 18 16:35:50.881195 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.880950    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt" event={"ID":"de443cc3-cf0a-452c-969f-a404401d7e8f","Type":"ContainerStarted","Data":"65121af6ff1886f7b4cb6b7e783dd45d2e195d5c80b0a9201100a993a79c9e24"}
Sep 18 16:35:50.911344 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:50.911289    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt" podStartSLOduration=0.911270829 podStartE2EDuration="911.270829ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:35:50.910648007 +0000 UTC m=+1846.819428385" watchObservedRunningTime="2026-09-18 16:35:50.911270829 +0000 UTC m=+1846.820051185"
Sep 18 16:35:55.087901 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:35:55.087862    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:36:00.489586 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:00.489542    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:36:00.490098 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:00.489606    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:36:00.502433 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:00.502410    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:36:00.957708 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:00.957675    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:36:05.099084 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:05.099055    2546 scope.go:117] "RemoveContainer" containerID="865328039ec4b9c51fa3b71f186343d5b1ca728213c3b2f1d858375e5dfad8f2"
Sep 18 16:36:05.107293 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:05.107273    2546 scope.go:117] "RemoveContainer" containerID="2f8ded875f281c5164260c7f5cbcab328c1c2e91bb6f667443200ae0d554fe42"
Sep 18 16:36:05.115323 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:05.115302    2546 scope.go:117] "RemoveContainer" containerID="73aff6d7c104064980b7284f3ffb5e00a5f9a7e19b67478fff872f5c6a6f346a"
Sep 18 16:36:14.221435 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.221400    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"]
Sep 18 16:36:14.225086 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.225061    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.229390 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.229165    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-11725e\"/\"kube-root-ca.crt\""
Sep 18 16:36:14.229390 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.229179    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-11725e\"/\"default-dockercfg-8bsfj\""
Sep 18 16:36:14.229390 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.229214    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-11725e\"/\"sub-test-model-11725e-kserve-self-signed-certs\""
Sep 18 16:36:14.229390 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.229315    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-11725e\"/\"openshift-service-ca.crt\""
Sep 18 16:36:14.242329 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.242299    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"]
Sep 18 16:36:14.378437 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.378397    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-dshm\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.378630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.378471    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-home\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.378630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.378521    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k292n\" (UniqueName: \"kubernetes.io/projected/47053e5a-808a-4d56-9b56-9524941908b0-kube-api-access-k292n\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.378630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.378552    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47053e5a-808a-4d56-9b56-9524941908b0-tls-certs\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.378630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.378584    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-tmp-dir\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.378850 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.378760    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-model-cache\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.480173 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.480072    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k292n\" (UniqueName: \"kubernetes.io/projected/47053e5a-808a-4d56-9b56-9524941908b0-kube-api-access-k292n\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.480173 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.480112    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47053e5a-808a-4d56-9b56-9524941908b0-tls-certs\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.480173 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.480149    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-tmp-dir\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.480479 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.480238    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-model-cache\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.480479 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.480315    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-dshm\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.480479 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.480361    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-home\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.480830 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.480803    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-tmp-dir\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.480948 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.480851    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-model-cache\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.480948 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.480879    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-home\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.482854 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.482824    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-dshm\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.482967 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.482899    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47053e5a-808a-4d56-9b56-9524941908b0-tls-certs\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.492275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.492250    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k292n\" (UniqueName: \"kubernetes.io/projected/47053e5a-808a-4d56-9b56-9524941908b0-kube-api-access-k292n\") pod \"sub-test-model-11725e-kserve-7c8cc67649-rr788\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") " pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.535608 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.535569    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:14.821445 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:14.821418    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"]
Sep 18 16:36:14.823742 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:36:14.823711    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47053e5a_808a_4d56_9b56_9524941908b0.slice/crio-514350a9d60b630389a5754a03c09917950dedef747a10a6c3b1a4c5ed9436c3 WatchSource:0}: Error finding container 514350a9d60b630389a5754a03c09917950dedef747a10a6c3b1a4c5ed9436c3: Status 404 returned error can't find the container with id 514350a9d60b630389a5754a03c09917950dedef747a10a6c3b1a4c5ed9436c3
Sep 18 16:36:15.037349 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:15.037307    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788" event={"ID":"47053e5a-808a-4d56-9b56-9524941908b0","Type":"ContainerStarted","Data":"78fea16b0165859e9638c1ad71b9c632d903eeb2521529c439b1ae1d2b56d917"}
Sep 18 16:36:15.037349 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:15.037355    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788" event={"ID":"47053e5a-808a-4d56-9b56-9524941908b0","Type":"ContainerStarted","Data":"514350a9d60b630389a5754a03c09917950dedef747a10a6c3b1a4c5ed9436c3"}
Sep 18 16:36:15.061385 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:15.061330    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788" podStartSLOduration=1.061312245 podStartE2EDuration="1.061312245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:36:15.058551927 +0000 UTC m=+1870.967332278" watchObservedRunningTime="2026-09-18 16:36:15.061312245 +0000 UTC m=+1870.970092595"
Sep 18 16:36:18.276361 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:18.276323    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:36:24.536264 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:24.536224    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:24.536264 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:24.536275    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:24.548820 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:24.548793    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:25.108228 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:25.108202    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:36:50.747146 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:36:50.747114    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:37:03.071330 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:03.071296    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7582-7f031f"]
Sep 18 16:37:03.075238 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:03.075216    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7582-7f031f"
Sep 18 16:37:03.079718 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:03.079695    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6q5gq\""
Sep 18 16:37:03.084852 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:03.084826    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7582-7f031f"]
Sep 18 16:37:03.253620 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:03.253579    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dlfg\" (UniqueName: \"kubernetes.io/projected/3ca1981d-8033-4854-9229-5d1535a050f0-kube-api-access-7dlfg\") pod \"test-curl-7582-7f031f\" (UID: \"3ca1981d-8033-4854-9229-5d1535a050f0\") " pod="opendatahub/test-curl-7582-7f031f"
Sep 18 16:37:03.354544 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:03.354444    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dlfg\" (UniqueName: \"kubernetes.io/projected/3ca1981d-8033-4854-9229-5d1535a050f0-kube-api-access-7dlfg\") pod \"test-curl-7582-7f031f\" (UID: \"3ca1981d-8033-4854-9229-5d1535a050f0\") " pod="opendatahub/test-curl-7582-7f031f"
Sep 18 16:37:03.365906 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:03.365875    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dlfg\" (UniqueName: \"kubernetes.io/projected/3ca1981d-8033-4854-9229-5d1535a050f0-kube-api-access-7dlfg\") pod \"test-curl-7582-7f031f\" (UID: \"3ca1981d-8033-4854-9229-5d1535a050f0\") " pod="opendatahub/test-curl-7582-7f031f"
Sep 18 16:37:03.386826 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:03.386780    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7582-7f031f"
Sep 18 16:37:03.661250 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:03.661225    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7582-7f031f"]
Sep 18 16:37:03.663003 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:37:03.662974    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ca1981d_8033_4854_9229_5d1535a050f0.slice/crio-3fe1148ea39ca75ea0b32bf7c155c5bc2551536d6ea3d360218c283c51024751 WatchSource:0}: Error finding container 3fe1148ea39ca75ea0b32bf7c155c5bc2551536d6ea3d360218c283c51024751: Status 404 returned error can't find the container with id 3fe1148ea39ca75ea0b32bf7c155c5bc2551536d6ea3d360218c283c51024751
Sep 18 16:37:04.320665 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:04.320629    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7582-7f031f" event={"ID":"3ca1981d-8033-4854-9229-5d1535a050f0","Type":"ContainerStarted","Data":"50ae1480265f7fe195ebd38f70177a1d4e1113b6da25891bac63dc44bbb2731a"}
Sep 18 16:37:04.321085 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:04.320678    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7582-7f031f" event={"ID":"3ca1981d-8033-4854-9229-5d1535a050f0","Type":"ContainerStarted","Data":"3fe1148ea39ca75ea0b32bf7c155c5bc2551536d6ea3d360218c283c51024751"}
Sep 18 16:37:04.342553 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:04.342502    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7582-7f031f" podStartSLOduration=1.3424845890000001 podStartE2EDuration="1.342484589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:37:04.340784351 +0000 UTC m=+1920.249564703" watchObservedRunningTime="2026-09-18 16:37:04.342484589 +0000 UTC m=+1920.251264941"
Sep 18 16:37:04.976534 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:04.976495    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7582-7f031f"]
Sep 18 16:37:04.983215 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:04.983183    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7582-7f031f"]
Sep 18 16:37:05.118141 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:05.118052    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7582-10b845"]
Sep 18 16:37:05.118452 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:05.118439    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca1981d-8033-4854-9229-5d1535a050f0" containerName="test-curl-7582-7f031f"
Sep 18 16:37:05.118452 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:05.118453    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca1981d-8033-4854-9229-5d1535a050f0" containerName="test-curl-7582-7f031f"
Sep 18 16:37:05.118549 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:05.118526    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca1981d-8033-4854-9229-5d1535a050f0" containerName="test-curl-7582-7f031f"
Sep 18 16:37:05.121694 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:05.121672    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7582-10b845"
Sep 18 16:37:05.141007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:05.140969    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7582-10b845"]
Sep 18 16:37:05.276516 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:05.276478    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bkrz\" (UniqueName: \"kubernetes.io/projected/4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249-kube-api-access-8bkrz\") pod \"test-curl-7582-10b845\" (UID: \"4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249\") " pod="opendatahub/test-curl-7582-10b845"
Sep 18 16:37:05.327297 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:37:05.327267    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7582-7f031f" container="test-curl-7582-7f031f"
Sep 18 16:37:05.377910 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:05.377813    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bkrz\" (UniqueName: \"kubernetes.io/projected/4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249-kube-api-access-8bkrz\") pod \"test-curl-7582-10b845\" (UID: \"4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249\") " pod="opendatahub/test-curl-7582-10b845"
Sep 18 16:37:05.393743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:05.393708    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bkrz\" (UniqueName: \"kubernetes.io/projected/4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249-kube-api-access-8bkrz\") pod \"test-curl-7582-10b845\" (UID: \"4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249\") " pod="opendatahub/test-curl-7582-10b845"
Sep 18 16:37:05.430237 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:05.430208    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7582-10b845"
Sep 18 16:37:05.723795 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:37:05.723761    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d51ed9b_5a7d_4f63_82ea_bcf6e67cb249.slice/crio-ef01b6363608749857cbc8305429e09a644c342bc12b28a345bc5c9b7d3e9e63 WatchSource:0}: Error finding container ef01b6363608749857cbc8305429e09a644c342bc12b28a345bc5c9b7d3e9e63: Status 404 returned error can't find the container with id ef01b6363608749857cbc8305429e09a644c342bc12b28a345bc5c9b7d3e9e63
Sep 18 16:37:05.727847 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:05.727820    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7582-10b845"]
Sep 18 16:37:06.334817 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:06.334770    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7582-10b845" event={"ID":"4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249","Type":"ContainerStarted","Data":"bfcbba714de89eeda545f0f2e900168a3b4fb09aa4c7105fd878a0914ab9f618"}
Sep 18 16:37:06.334817 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:06.334824    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7582-10b845" event={"ID":"4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249","Type":"ContainerStarted","Data":"ef01b6363608749857cbc8305429e09a644c342bc12b28a345bc5c9b7d3e9e63"}
Sep 18 16:37:06.335351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:06.334820    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7582-7f031f" podUID="3ca1981d-8033-4854-9229-5d1535a050f0" containerName="test-curl-7582-7f031f" containerID="cri-o://50ae1480265f7fe195ebd38f70177a1d4e1113b6da25891bac63dc44bbb2731a" gracePeriod=2
Sep 18 16:37:06.340509 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:37:06.340483    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7582-7f031f\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="3ca1981d-8033-4854-9229-5d1535a050f0" pod="opendatahub/test-curl-7582-7f031f"
Sep 18 16:37:06.366308 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:06.366253    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7582-10b845" podStartSLOduration=1.36623716 podStartE2EDuration="1.36623716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:37:06.363876066 +0000 UTC m=+1922.272656418" watchObservedRunningTime="2026-09-18 16:37:06.36623716 +0000 UTC m=+1922.275017561"
Sep 18 16:37:06.366859 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:37:06.366826    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7582-7f031f\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="3ca1981d-8033-4854-9229-5d1535a050f0" pod="opendatahub/test-curl-7582-7f031f"
Sep 18 16:37:06.674560 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:06.671086    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:37:07.026278 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:07.026241    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7582-10b845"]
Sep 18 16:37:07.029445 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:07.029418    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7582-10b845"]
Sep 18 16:37:07.340573 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:37:07.340489    2546 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7582-10b845" container="test-curl-7582-10b845"
Sep 18 16:37:08.346581 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:08.346540    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7582-10b845" podUID="4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249" containerName="test-curl-7582-10b845" containerID="cri-o://bfcbba714de89eeda545f0f2e900168a3b4fb09aa4c7105fd878a0914ab9f618" gracePeriod=2
Sep 18 16:37:08.353071 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:37:08.353014    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7582-10b845\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249" pod="opendatahub/test-curl-7582-10b845"
Sep 18 16:37:08.528165 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:08.528132    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7582-7f031f"
Sep 18 16:37:08.530698 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:37:08.530665    2546 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7582-10b845\" is forbidden: User \"system:node:ip-10-0-135-253.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-253.ec2.internal' and this object" podUID="4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249" pod="opendatahub/test-curl-7582-10b845"
Sep 18 16:37:08.705304 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:08.705276    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dlfg\" (UniqueName: \"kubernetes.io/projected/3ca1981d-8033-4854-9229-5d1535a050f0-kube-api-access-7dlfg\") pod \"3ca1981d-8033-4854-9229-5d1535a050f0\" (UID: \"3ca1981d-8033-4854-9229-5d1535a050f0\") "
Sep 18 16:37:08.707399 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:08.707378    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca1981d-8033-4854-9229-5d1535a050f0-kube-api-access-7dlfg" (OuterVolumeSpecName: "kube-api-access-7dlfg") pod "3ca1981d-8033-4854-9229-5d1535a050f0" (UID: "3ca1981d-8033-4854-9229-5d1535a050f0"). InnerVolumeSpecName "kube-api-access-7dlfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:37:08.806068 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:08.805973    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dlfg\" (UniqueName: \"kubernetes.io/projected/3ca1981d-8033-4854-9229-5d1535a050f0-kube-api-access-7dlfg\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:09.147373 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:09.147276    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"]
Sep 18 16:37:09.353599 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:09.353562    2546 generic.go:356] "Generic (PLEG): container finished" podID="3ca1981d-8033-4854-9229-5d1535a050f0" containerID="50ae1480265f7fe195ebd38f70177a1d4e1113b6da25891bac63dc44bbb2731a" exitCode=137
Sep 18 16:37:09.353599 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:09.353601    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7582-7f031f"
Sep 18 16:37:09.354124 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:09.353659    2546 scope.go:117] "RemoveContainer" containerID="50ae1480265f7fe195ebd38f70177a1d4e1113b6da25891bac63dc44bbb2731a"
Sep 18 16:37:09.362998 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:09.362981    2546 scope.go:117] "RemoveContainer" containerID="50ae1480265f7fe195ebd38f70177a1d4e1113b6da25891bac63dc44bbb2731a"
Sep 18 16:37:09.363322 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:37:09.363304    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ae1480265f7fe195ebd38f70177a1d4e1113b6da25891bac63dc44bbb2731a\": container with ID starting with 50ae1480265f7fe195ebd38f70177a1d4e1113b6da25891bac63dc44bbb2731a not found: ID does not exist" containerID="50ae1480265f7fe195ebd38f70177a1d4e1113b6da25891bac63dc44bbb2731a"
Sep 18 16:37:09.363370 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:09.363332    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ae1480265f7fe195ebd38f70177a1d4e1113b6da25891bac63dc44bbb2731a"} err="failed to get container status \"50ae1480265f7fe195ebd38f70177a1d4e1113b6da25891bac63dc44bbb2731a\": rpc error: code = NotFound desc = could not find container \"50ae1480265f7fe195ebd38f70177a1d4e1113b6da25891bac63dc44bbb2731a\": container with ID starting with 50ae1480265f7fe195ebd38f70177a1d4e1113b6da25891bac63dc44bbb2731a not found: ID does not exist"
Sep 18 16:37:10.248333 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:10.248300    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"]
Sep 18 16:37:10.510671 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:10.510595    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7582-10b845"
Sep 18 16:37:10.520958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:10.520929    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bkrz\" (UniqueName: \"kubernetes.io/projected/4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249-kube-api-access-8bkrz\") pod \"4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249\" (UID: \"4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249\") "
Sep 18 16:37:10.523010 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:10.522984    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249-kube-api-access-8bkrz" (OuterVolumeSpecName: "kube-api-access-8bkrz") pod "4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249" (UID: "4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249"). InnerVolumeSpecName "kube-api-access-8bkrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:37:10.603432 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:10.603398    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca1981d-8033-4854-9229-5d1535a050f0" path="/var/lib/kubelet/pods/3ca1981d-8033-4854-9229-5d1535a050f0/volumes"
Sep 18 16:37:10.603755 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:10.603743    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249" path="/var/lib/kubelet/pods/4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249/volumes"
Sep 18 16:37:10.622538 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:10.622508    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bkrz\" (UniqueName: \"kubernetes.io/projected/4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249-kube-api-access-8bkrz\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:11.368053 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:11.367988    2546 generic.go:356] "Generic (PLEG): container finished" podID="4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249" containerID="bfcbba714de89eeda545f0f2e900168a3b4fb09aa4c7105fd878a0914ab9f618" exitCode=137
Sep 18 16:37:11.368306 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:11.368086    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7582-10b845"
Sep 18 16:37:11.368306 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:11.368098    2546 scope.go:117] "RemoveContainer" containerID="bfcbba714de89eeda545f0f2e900168a3b4fb09aa4c7105fd878a0914ab9f618"
Sep 18 16:37:11.377007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:11.376987    2546 scope.go:117] "RemoveContainer" containerID="bfcbba714de89eeda545f0f2e900168a3b4fb09aa4c7105fd878a0914ab9f618"
Sep 18 16:37:11.377481 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:37:11.377458    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfcbba714de89eeda545f0f2e900168a3b4fb09aa4c7105fd878a0914ab9f618\": container with ID starting with bfcbba714de89eeda545f0f2e900168a3b4fb09aa4c7105fd878a0914ab9f618 not found: ID does not exist" containerID="bfcbba714de89eeda545f0f2e900168a3b4fb09aa4c7105fd878a0914ab9f618"
Sep 18 16:37:11.377554 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:11.377491    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfcbba714de89eeda545f0f2e900168a3b4fb09aa4c7105fd878a0914ab9f618"} err="failed to get container status \"bfcbba714de89eeda545f0f2e900168a3b4fb09aa4c7105fd878a0914ab9f618\": rpc error: code = NotFound desc = could not find container \"bfcbba714de89eeda545f0f2e900168a3b4fb09aa4c7105fd878a0914ab9f618\": container with ID starting with bfcbba714de89eeda545f0f2e900168a3b4fb09aa4c7105fd878a0914ab9f618 not found: ID does not exist"
Sep 18 16:37:19.154287 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:19.154250    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:37:24.166648 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.166582    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt" podUID="de443cc3-cf0a-452c-969f-a404401d7e8f" containerName="main" containerID="cri-o://02c7e72e4a55b444d56f237b35c7cb7f012983159be03f5fe3493e8e5fb63cfb" gracePeriod=45
Sep 18 16:37:24.436941 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.436909    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:37:24.450105 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.450065    2546 generic.go:356] "Generic (PLEG): container finished" podID="de443cc3-cf0a-452c-969f-a404401d7e8f" containerID="02c7e72e4a55b444d56f237b35c7cb7f012983159be03f5fe3493e8e5fb63cfb" exitCode=0
Sep 18 16:37:24.450275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.450130    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"
Sep 18 16:37:24.450275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.450131    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt" event={"ID":"de443cc3-cf0a-452c-969f-a404401d7e8f","Type":"ContainerDied","Data":"02c7e72e4a55b444d56f237b35c7cb7f012983159be03f5fe3493e8e5fb63cfb"}
Sep 18 16:37:24.450275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.450179    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt" event={"ID":"de443cc3-cf0a-452c-969f-a404401d7e8f","Type":"ContainerDied","Data":"65121af6ff1886f7b4cb6b7e783dd45d2e195d5c80b0a9201100a993a79c9e24"}
Sep 18 16:37:24.450275 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.450197    2546 scope.go:117] "RemoveContainer" containerID="02c7e72e4a55b444d56f237b35c7cb7f012983159be03f5fe3493e8e5fb63cfb"
Sep 18 16:37:24.459780 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.459747    2546 scope.go:117] "RemoveContainer" containerID="02c7e72e4a55b444d56f237b35c7cb7f012983159be03f5fe3493e8e5fb63cfb"
Sep 18 16:37:24.460077 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:37:24.460058    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c7e72e4a55b444d56f237b35c7cb7f012983159be03f5fe3493e8e5fb63cfb\": container with ID starting with 02c7e72e4a55b444d56f237b35c7cb7f012983159be03f5fe3493e8e5fb63cfb not found: ID does not exist" containerID="02c7e72e4a55b444d56f237b35c7cb7f012983159be03f5fe3493e8e5fb63cfb"
Sep 18 16:37:24.460155 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.460085    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c7e72e4a55b444d56f237b35c7cb7f012983159be03f5fe3493e8e5fb63cfb"} err="failed to get container status \"02c7e72e4a55b444d56f237b35c7cb7f012983159be03f5fe3493e8e5fb63cfb\": rpc error: code = NotFound desc = could not find container \"02c7e72e4a55b444d56f237b35c7cb7f012983159be03f5fe3493e8e5fb63cfb\": container with ID starting with 02c7e72e4a55b444d56f237b35c7cb7f012983159be03f5fe3493e8e5fb63cfb not found: ID does not exist"
Sep 18 16:37:24.574056 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.573991    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l49s5\" (UniqueName: \"kubernetes.io/projected/de443cc3-cf0a-452c-969f-a404401d7e8f-kube-api-access-l49s5\") pod \"de443cc3-cf0a-452c-969f-a404401d7e8f\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") "
Sep 18 16:37:24.574230 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.574078    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-home\") pod \"de443cc3-cf0a-452c-969f-a404401d7e8f\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") "
Sep 18 16:37:24.574230 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.574122    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de443cc3-cf0a-452c-969f-a404401d7e8f-tls-certs\") pod \"de443cc3-cf0a-452c-969f-a404401d7e8f\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") "
Sep 18 16:37:24.574230 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.574173    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-dshm\") pod \"de443cc3-cf0a-452c-969f-a404401d7e8f\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") "
Sep 18 16:37:24.574230 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.574197    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-tmp-dir\") pod \"de443cc3-cf0a-452c-969f-a404401d7e8f\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") "
Sep 18 16:37:24.574486 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.574234    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-model-cache\") pod \"de443cc3-cf0a-452c-969f-a404401d7e8f\" (UID: \"de443cc3-cf0a-452c-969f-a404401d7e8f\") "
Sep 18 16:37:24.574486 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.574325    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-home" (OuterVolumeSpecName: "home") pod "de443cc3-cf0a-452c-969f-a404401d7e8f" (UID: "de443cc3-cf0a-452c-969f-a404401d7e8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:37:24.574596 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.574505    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de443cc3-cf0a-452c-969f-a404401d7e8f" (UID: "de443cc3-cf0a-452c-969f-a404401d7e8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:37:24.574596 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.574533    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-model-cache" (OuterVolumeSpecName: "model-cache") pod "de443cc3-cf0a-452c-969f-a404401d7e8f" (UID: "de443cc3-cf0a-452c-969f-a404401d7e8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:37:24.574727 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.574690    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:24.574727 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.574707    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:24.574727 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.574719    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:24.576509 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.576487    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de443cc3-cf0a-452c-969f-a404401d7e8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de443cc3-cf0a-452c-969f-a404401d7e8f" (UID: "de443cc3-cf0a-452c-969f-a404401d7e8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:37:24.576616 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.576487    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de443cc3-cf0a-452c-969f-a404401d7e8f-kube-api-access-l49s5" (OuterVolumeSpecName: "kube-api-access-l49s5") pod "de443cc3-cf0a-452c-969f-a404401d7e8f" (UID: "de443cc3-cf0a-452c-969f-a404401d7e8f"). InnerVolumeSpecName "kube-api-access-l49s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:37:24.576616 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.576486    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-dshm" (OuterVolumeSpecName: "dshm") pod "de443cc3-cf0a-452c-969f-a404401d7e8f" (UID: "de443cc3-cf0a-452c-969f-a404401d7e8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:37:24.675696 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.675602    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de443cc3-cf0a-452c-969f-a404401d7e8f-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:24.675696 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.675640    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de443cc3-cf0a-452c-969f-a404401d7e8f-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:24.675696 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.675650    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l49s5\" (UniqueName: \"kubernetes.io/projected/de443cc3-cf0a-452c-969f-a404401d7e8f-kube-api-access-l49s5\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:24.771373 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.771330    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"]
Sep 18 16:37:24.782215 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:24.782185    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/sub-test-model-a75117-kserve-76c4d85cf5-wfprt"]
Sep 18 16:37:25.268163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.268119    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788" podUID="47053e5a-808a-4d56-9b56-9524941908b0" containerName="main" containerID="cri-o://78fea16b0165859e9638c1ad71b9c632d903eeb2521529c439b1ae1d2b56d917" gracePeriod=45
Sep 18 16:37:25.458772 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.458737    2546 generic.go:356] "Generic (PLEG): container finished" podID="47053e5a-808a-4d56-9b56-9524941908b0" containerID="78fea16b0165859e9638c1ad71b9c632d903eeb2521529c439b1ae1d2b56d917" exitCode=0
Sep 18 16:37:25.458935 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.458805    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788" event={"ID":"47053e5a-808a-4d56-9b56-9524941908b0","Type":"ContainerDied","Data":"78fea16b0165859e9638c1ad71b9c632d903eeb2521529c439b1ae1d2b56d917"}
Sep 18 16:37:25.537143 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.537070    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:37:25.687443 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.687409    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-model-cache\") pod \"47053e5a-808a-4d56-9b56-9524941908b0\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") "
Sep 18 16:37:25.687618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.687473    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k292n\" (UniqueName: \"kubernetes.io/projected/47053e5a-808a-4d56-9b56-9524941908b0-kube-api-access-k292n\") pod \"47053e5a-808a-4d56-9b56-9524941908b0\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") "
Sep 18 16:37:25.687618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.687513    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47053e5a-808a-4d56-9b56-9524941908b0-tls-certs\") pod \"47053e5a-808a-4d56-9b56-9524941908b0\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") "
Sep 18 16:37:25.687618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.687548    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-tmp-dir\") pod \"47053e5a-808a-4d56-9b56-9524941908b0\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") "
Sep 18 16:37:25.687618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.687614    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-dshm\") pod \"47053e5a-808a-4d56-9b56-9524941908b0\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") "
Sep 18 16:37:25.687759 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.687627    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "47053e5a-808a-4d56-9b56-9524941908b0" (UID: "47053e5a-808a-4d56-9b56-9524941908b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:37:25.687759 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.687663    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-home\") pod \"47053e5a-808a-4d56-9b56-9524941908b0\" (UID: \"47053e5a-808a-4d56-9b56-9524941908b0\") "
Sep 18 16:37:25.687887 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.687867    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "47053e5a-808a-4d56-9b56-9524941908b0" (UID: "47053e5a-808a-4d56-9b56-9524941908b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:37:25.687972 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.687950    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-home" (OuterVolumeSpecName: "home") pod "47053e5a-808a-4d56-9b56-9524941908b0" (UID: "47053e5a-808a-4d56-9b56-9524941908b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:37:25.688221 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.688159    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:25.688221 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.688183    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:25.688221 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.688196    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:25.689935 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.689914    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-dshm" (OuterVolumeSpecName: "dshm") pod "47053e5a-808a-4d56-9b56-9524941908b0" (UID: "47053e5a-808a-4d56-9b56-9524941908b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:37:25.690192 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.690165    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47053e5a-808a-4d56-9b56-9524941908b0-kube-api-access-k292n" (OuterVolumeSpecName: "kube-api-access-k292n") pod "47053e5a-808a-4d56-9b56-9524941908b0" (UID: "47053e5a-808a-4d56-9b56-9524941908b0"). InnerVolumeSpecName "kube-api-access-k292n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:37:25.690192 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.690177    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47053e5a-808a-4d56-9b56-9524941908b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "47053e5a-808a-4d56-9b56-9524941908b0" (UID: "47053e5a-808a-4d56-9b56-9524941908b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:37:25.789353 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.789250    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47053e5a-808a-4d56-9b56-9524941908b0-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:25.789353 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.789285    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k292n\" (UniqueName: \"kubernetes.io/projected/47053e5a-808a-4d56-9b56-9524941908b0-kube-api-access-k292n\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:25.789353 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:25.789296    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47053e5a-808a-4d56-9b56-9524941908b0-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:37:26.466941 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:26.466911    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"
Sep 18 16:37:26.466941 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:26.466928    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788" event={"ID":"47053e5a-808a-4d56-9b56-9524941908b0","Type":"ContainerDied","Data":"514350a9d60b630389a5754a03c09917950dedef747a10a6c3b1a4c5ed9436c3"}
Sep 18 16:37:26.467467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:26.466978    2546 scope.go:117] "RemoveContainer" containerID="78fea16b0165859e9638c1ad71b9c632d903eeb2521529c439b1ae1d2b56d917"
Sep 18 16:37:26.505723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:26.505684    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"]
Sep 18 16:37:26.518282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:26.518249    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-11725e/sub-test-model-11725e-kserve-7c8cc67649-rr788"]
Sep 18 16:37:26.604422 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:26.604393    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47053e5a-808a-4d56-9b56-9524941908b0" path="/var/lib/kubelet/pods/47053e5a-808a-4d56-9b56-9524941908b0/volumes"
Sep 18 16:37:26.604780 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:37:26.604768    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de443cc3-cf0a-452c-969f-a404401d7e8f" path="/var/lib/kubelet/pods/de443cc3-cf0a-452c-969f-a404401d7e8f/volumes"
Sep 18 16:38:01.662256 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:01.662215    2546 generic.go:356] "Generic (PLEG): container finished" podID="d9b87cf9-1252-4eeb-bd58-194c81ea4a94" containerID="38dcdd6f151f25ff14e35fca9d69d099ae915f1b50af9d604fb1bb1917144906" exitCode=1
Sep 18 16:38:01.662792 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:01.662293    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" event={"ID":"d9b87cf9-1252-4eeb-bd58-194c81ea4a94","Type":"ContainerDied","Data":"38dcdd6f151f25ff14e35fca9d69d099ae915f1b50af9d604fb1bb1917144906"}
Sep 18 16:38:01.662792 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:01.662756    2546 scope.go:117] "RemoveContainer" containerID="38dcdd6f151f25ff14e35fca9d69d099ae915f1b50af9d604fb1bb1917144906"
Sep 18 16:38:01.663158 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:01.663140    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:38:02.669542 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:02.669507    2546 generic.go:356] "Generic (PLEG): container finished" podID="61d67412-30af-4fb0-adc0-5251f88b7a41" containerID="324122dd9d31d30f4c518b31ef077ad5febc35257d54ba486e28086f31a52233" exitCode=1
Sep 18 16:38:02.669893 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:02.669580    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" event={"ID":"61d67412-30af-4fb0-adc0-5251f88b7a41","Type":"ContainerDied","Data":"324122dd9d31d30f4c518b31ef077ad5febc35257d54ba486e28086f31a52233"}
Sep 18 16:38:02.669994 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:02.669978    2546 scope.go:117] "RemoveContainer" containerID="324122dd9d31d30f4c518b31ef077ad5febc35257d54ba486e28086f31a52233"
Sep 18 16:38:02.671737 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:02.671710    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" event={"ID":"d9b87cf9-1252-4eeb-bd58-194c81ea4a94","Type":"ContainerStarted","Data":"7ffa2bf0a0351755ec4c7d243d09d324b5c16d217e2986a9043f161fd394f65d"}
Sep 18 16:38:02.671936 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:02.671917    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:38:02.673723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:02.673706    2546 generic.go:356] "Generic (PLEG): container finished" podID="f7e45976-cc30-418c-96f4-70e0c667572e" containerID="c9e1290130d2c65bb0fe3bcbf5b1c61eec051bc0cb3273a15f51a611fb97a9ff" exitCode=1
Sep 18 16:38:02.673797 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:02.673777    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" event={"ID":"f7e45976-cc30-418c-96f4-70e0c667572e","Type":"ContainerDied","Data":"c9e1290130d2c65bb0fe3bcbf5b1c61eec051bc0cb3273a15f51a611fb97a9ff"}
Sep 18 16:38:02.674145 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:02.674129    2546 scope.go:117] "RemoveContainer" containerID="c9e1290130d2c65bb0fe3bcbf5b1c61eec051bc0cb3273a15f51a611fb97a9ff"
Sep 18 16:38:03.451303 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.451258    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/healthz\": dial tcp 10.133.0.34:8081: connect: connection refused"
Sep 18 16:38:03.520258 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.520208    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:38:03.680963 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.680930    2546 generic.go:356] "Generic (PLEG): container finished" podID="4ce26bbe-bd40-4653-a16f-2b0c1c87604b" containerID="536ee85a2a92b0402144cad8b3453f6dd54c4a74f49bcf0a141ee2c7661b28cd" exitCode=1
Sep 18 16:38:03.681400 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.681004    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" event={"ID":"4ce26bbe-bd40-4653-a16f-2b0c1c87604b","Type":"ContainerDied","Data":"536ee85a2a92b0402144cad8b3453f6dd54c4a74f49bcf0a141ee2c7661b28cd"}
Sep 18 16:38:03.681485 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.681408    2546 scope.go:117] "RemoveContainer" containerID="536ee85a2a92b0402144cad8b3453f6dd54c4a74f49bcf0a141ee2c7661b28cd"
Sep 18 16:38:03.683469 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.683446    2546 generic.go:356] "Generic (PLEG): container finished" podID="1c53eeff-c125-49c6-ab26-182dd935f398" containerID="df9864ed5a6bb54db9cd5a5cd3f4d5c4751b13a3865a1f7de0a0cfc58b6b2d0c" exitCode=0
Sep 18 16:38:03.683597 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.683516    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" event={"ID":"1c53eeff-c125-49c6-ab26-182dd935f398","Type":"ContainerDied","Data":"df9864ed5a6bb54db9cd5a5cd3f4d5c4751b13a3865a1f7de0a0cfc58b6b2d0c"}
Sep 18 16:38:03.683597 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.683575    2546 scope.go:117] "RemoveContainer" containerID="d7b66f6dcf15f0ee940b6003c3a51567213d18789e0c372e06cdd646e21ee788"
Sep 18 16:38:03.683930 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.683913    2546 scope.go:117] "RemoveContainer" containerID="df9864ed5a6bb54db9cd5a5cd3f4d5c4751b13a3865a1f7de0a0cfc58b6b2d0c"
Sep 18 16:38:03.684213 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:38:03.684179    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-l7p9n_kuadrant-system(1c53eeff-c125-49c6-ab26-182dd935f398)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podUID="1c53eeff-c125-49c6-ab26-182dd935f398"
Sep 18 16:38:03.685875 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.685849    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" event={"ID":"61d67412-30af-4fb0-adc0-5251f88b7a41","Type":"ContainerStarted","Data":"c84ba3c21c11d4dc01d7be3d0106b61695f39004e266889e7d87594d1f65f2e0"}
Sep 18 16:38:03.685997 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.685914    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:38:03.687460 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.687442    2546 generic.go:356] "Generic (PLEG): container finished" podID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerID="778bb1ef5585ff103e7dc27354d8781cc8f5383595cafc9fccc126ca3c7970dd" exitCode=1
Sep 18 16:38:03.687546 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.687508    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerDied","Data":"778bb1ef5585ff103e7dc27354d8781cc8f5383595cafc9fccc126ca3c7970dd"}
Sep 18 16:38:03.687845 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.687825    2546 scope.go:117] "RemoveContainer" containerID="778bb1ef5585ff103e7dc27354d8781cc8f5383595cafc9fccc126ca3c7970dd"
Sep 18 16:38:03.689821 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.689800    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" event={"ID":"f7e45976-cc30-418c-96f4-70e0c667572e","Type":"ContainerStarted","Data":"b7334fbd6b0e98847e2e4fb07a38bb00c701eccddfa90666692ae04e16f6f467"}
Sep 18 16:38:03.690079 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:03.690065    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:38:04.240151 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:04.240118    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:38:04.410092 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:04.409958    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:38:04.696721 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:04.696683    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" event={"ID":"4ce26bbe-bd40-4653-a16f-2b0c1c87604b","Type":"ContainerStarted","Data":"082ac3710cb649a00e0130d32f7918b311fcb6280e05340bab5e2d2ec192fe22"}
Sep 18 16:38:04.697141 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:04.696911    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:38:04.700244 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:04.700215    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerStarted","Data":"ec7a0640f0844f3a40d039c2f480dfcb228250877b0dee0820cd1c757acd9233"}
Sep 18 16:38:04.700383 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:04.700283    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:38:05.684109 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:05.684068    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:38:05.684468 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:05.684453    2546 scope.go:117] "RemoveContainer" containerID="df9864ed5a6bb54db9cd5a5cd3f4d5c4751b13a3865a1f7de0a0cfc58b6b2d0c"
Sep 18 16:38:05.684642 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:38:05.684626    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-l7p9n_kuadrant-system(1c53eeff-c125-49c6-ab26-182dd935f398)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podUID="1c53eeff-c125-49c6-ab26-182dd935f398"
Sep 18 16:38:14.702796 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:14.702765    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:38:14.703290 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:14.702847    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:38:15.707590 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:15.707554    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:38:19.696840 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:38:19.696793    2546 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:38:20.596873 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:20.596840    2546 scope.go:117] "RemoveContainer" containerID="df9864ed5a6bb54db9cd5a5cd3f4d5c4751b13a3865a1f7de0a0cfc58b6b2d0c"
Sep 18 16:38:20.789477 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:20.789441    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" event={"ID":"1c53eeff-c125-49c6-ab26-182dd935f398","Type":"ContainerStarted","Data":"91052c7648f4751de2d6422199ec5006f72203e0c0d6f23b205f796abdc7e95f"}
Sep 18 16:38:20.789841 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:20.789649    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:38:31.798059 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:31.798002    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:38:33.692373 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:33.692339    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:38:35.668571 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:38:35.668470    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{llmisvc-controller-manager-777c6bd7d5-8bfgv.18d67641767bb5fc  opendatahub   14841 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:llmisvc-controller-manager-777c6bd7d5-8bfgv,UID:d9b87cf9-1252-4eeb-bd58-194c81ea4a94,APIVersion:v1,ResourceVersion:14762,FieldPath:spec.containers{manager},},Reason:Pulling,Message:Pulling image \"quay.io/opendatahub/odh-kserve-llmisvc-controller:odh-v3.5-EA2\",Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:12:05 +0000 UTC,LastTimestamp:2026-09-18 16:38:01.663329607 +0000 UTC m=+1977.572109941,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:38:35.709343 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:38:35.709314    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:38:46.319995 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:38:46.319956    2546 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:38:46.407688 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:38:46.407649    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:36Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:36Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:36Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:38:36Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-135-253.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal/status?timeout=10s\": context deadline exceeded"
Sep 18 16:38:56.320543 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:38:56.320477    2546 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:38:56.408686 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:38:56.408641    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:39:01.666114 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:01.666061    2546 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods llmisvc-controller-manager-777c6bd7d5-8bfgv)" podUID="d9b87cf9-1252-4eeb-bd58-194c81ea4a94" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:39:05.042360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:39:05.042263    2546 generic.go:356] "Generic (PLEG): container finished" podID="5d260854-98c0-4dd7-89a4-d9b9609d3203" containerID="0bb3a863b4ce424950b1c22a41c6f615246bbf230234136b2d2e1e4256e5cc42" exitCode=1
Sep 18 16:39:05.042360 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:39:05.042340    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerDied","Data":"0bb3a863b4ce424950b1c22a41c6f615246bbf230234136b2d2e1e4256e5cc42"}
Sep 18 16:39:05.042776 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:39:05.042685    2546 scope.go:117] "RemoveContainer" containerID="0bb3a863b4ce424950b1c22a41c6f615246bbf230234136b2d2e1e4256e5cc42"
Sep 18 16:39:05.181100 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:39:05.181071    2546 scope.go:117] "RemoveContainer" containerID="3a5142d1429b851e5f67019df6d4c45a516d8cef431a084044cdf4059fe2a5ef"
Sep 18 16:39:05.191462 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:39:05.191432    2546 scope.go:117] "RemoveContainer" containerID="248c8ada99764c09e340f92e5694d978b2b8011c932ad266dd074513468ff4e2"
Sep 18 16:39:06.050398 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:39:06.050362    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerStarted","Data":"71f41230ea0c43f34fec260fa25087faee67f6ac34948b53875ff6673fac4a47"}
Sep 18 16:39:06.321293 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:06.321188    2546 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:39:06.409181 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:06.409144    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:39:09.672448 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:09.672348    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{llmisvc-controller-manager-777c6bd7d5-8bfgv.18d677abe8dfae28  opendatahub    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:llmisvc-controller-manager-777c6bd7d5-8bfgv,UID:d9b87cf9-1252-4eeb-bd58-194c81ea4a94,APIVersion:v1,ResourceVersion:14762,FieldPath:spec.containers{manager},},Reason:Pulled,Message:Successfully pulled image \"quay.io/opendatahub/odh-kserve-llmisvc-controller:odh-v3.5-EA2\" in 493ms (493ms including waiting). Image size: 133973437 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:38:02.156437032 +0000 UTC m=+1978.065217362,LastTimestamp:2026-09-18 16:38:02.156437032 +0000 UTC m=+1978.065217362,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:39:16.321847 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:16.321793    2546 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:39:16.410176 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:16.410137    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:39:26.323086 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:26.323039    2546 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:39:26.323086 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:39:26.323088    2546 controller.go:115] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease"
Sep 18 16:39:26.411348 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:26.411309    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:39:26.411348 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:26.411339    2546 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Sep 18 16:39:36.323840 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:36.323741    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="200ms"
Sep 18 16:39:43.675736 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:43.675639    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{llmisvc-controller-manager-777c6bd7d5-8bfgv.18d6764230b9444f  opendatahub   14918 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:llmisvc-controller-manager-777c6bd7d5-8bfgv,UID:d9b87cf9-1252-4eeb-bd58-194c81ea4a94,APIVersion:v1,ResourceVersion:14762,FieldPath:spec.containers{manager},},Reason:Created,Message:Created container: manager,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:12:08 +0000 UTC,LastTimestamp:2026-09-18 16:38:02.287980058 +0000 UTC m=+1978.196760400,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:39:46.479360 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:46.479310    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:39:36Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:39:36Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:39:36Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:39:36Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-135-253.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal/status?timeout=10s\": context deadline exceeded"
Sep 18 16:39:46.525133 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:46.525093    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="400ms"
Sep 18 16:39:56.480332 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:56.480276    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:39:56.926276 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:39:56.926175    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded" interval="800ms"
Sep 18 16:40:01.670146 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:40:01.670101    2546 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods limitador-operator-controller-manager-74495c69f8-dbsdv)" podUID="61d67412-30af-4fb0-adc0-5251f88b7a41" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:40:05.215745 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:40:05.215712    2546 scope.go:117] "RemoveContainer" containerID="f360032c042b727c15af3dd440cbc8b5c3d01cdd1ddbc6e8b4b5906053e217cd"
Sep 18 16:40:05.246591 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:40:05.246565    2546 scope.go:117] "RemoveContainer" containerID="6d7f242db4d22d4ca9e80c02e4c743b34d288b370b15b4c34217585428f68ad6"
Sep 18 16:40:05.254804 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:40:05.254781    2546 scope.go:117] "RemoveContainer" containerID="be7015b02006302cd4856bd647e5999f7111a5ef9f83d9e9a71ea9639490aaba"
Sep 18 16:40:05.392250 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:40:05.392215    2546 generic.go:356] "Generic (PLEG): container finished" podID="5d260854-98c0-4dd7-89a4-d9b9609d3203" containerID="71f41230ea0c43f34fec260fa25087faee67f6ac34948b53875ff6673fac4a47" exitCode=1
Sep 18 16:40:05.392424 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:40:05.392290    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerDied","Data":"71f41230ea0c43f34fec260fa25087faee67f6ac34948b53875ff6673fac4a47"}
Sep 18 16:40:05.392424 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:40:05.392353    2546 scope.go:117] "RemoveContainer" containerID="0bb3a863b4ce424950b1c22a41c6f615246bbf230234136b2d2e1e4256e5cc42"
Sep 18 16:40:05.392717 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:40:05.392687    2546 scope.go:117] "RemoveContainer" containerID="71f41230ea0c43f34fec260fa25087faee67f6ac34948b53875ff6673fac4a47"
Sep 18 16:40:05.392917 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:40:05.392895    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:40:06.480831 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:40:06.480794    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:40:07.727535 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:40:07.727477    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="1.6s"
Sep 18 16:40:16.481756 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:40:16.481712    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:40:17.679469 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:40:17.679374    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{llmisvc-controller-manager-777c6bd7d5-8bfgv.18d67642311c4d93  opendatahub   14919 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:llmisvc-controller-manager-777c6bd7d5-8bfgv,UID:d9b87cf9-1252-4eeb-bd58-194c81ea4a94,APIVersion:v1,ResourceVersion:14762,FieldPath:spec.containers{manager},},Reason:Started,Message:Started container manager,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:12:08 +0000 UTC,LastTimestamp:2026-09-18 16:38:02.295716436 +0000 UTC m=+1978.204496788,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:40:19.328993 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:40:19.328926    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded" interval="3.2s"
Sep 18 16:40:19.597391 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:40:19.597305    2546 scope.go:117] "RemoveContainer" containerID="71f41230ea0c43f34fec260fa25087faee67f6ac34948b53875ff6673fac4a47"
Sep 18 16:40:20.481264 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:40:20.481227    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerStarted","Data":"4515a05e11c422c5e459cd747ae4a4bb6108fdfed150fc766e5529f7d4054c51"}
Sep 18 16:40:26.482245 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:40:26.482201    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:40:26.482245 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:40:26.482237    2546 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Sep 18 16:40:32.530805 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:40:32.530755    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="6.4s"
Sep 18 16:40:48.932328 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:40:48.932280    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:40:51.683244 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:40:51.683146    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{limitador-operator-controller-manager-74495c69f8-dbsdv.18d676585dcd041a  kuadrant-system   18813 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kuadrant-system,Name:limitador-operator-controller-manager-74495c69f8-dbsdv,UID:61d67412-30af-4fb0-adc0-5251f88b7a41,APIVersion:v1,ResourceVersion:18696,FieldPath:spec.containers{manager},},Reason:Pulled,Message:Container image \"registry.redhat.io/rhcl-1/limitador-rhel9-operator@sha256:7fdc92b22c7d74e68e71ebf54d1c680fc8dbd47198286dc4f7f87b3d4b31472e\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:13:43 +0000 UTC,LastTimestamp:2026-09-18 16:38:02.670878903 +0000 UTC m=+1978.579659248,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:41:01.449914 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:01.449813    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:40:51Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:40:51Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:40:51Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:40:51Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-135-253.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal/status?timeout=10s\": context deadline exceeded"
Sep 18 16:41:05.281703 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:05.281668    2546 scope.go:117] "RemoveContainer" containerID="97c8710df4169a24d9bedec7721b88847cbaf6c852af080fd57266f0049ab890"
Sep 18 16:41:05.933631 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:05.933585    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:41:11.450511 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:11.450478    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:41:13.764066 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:13.764004    2546 generic.go:356] "Generic (PLEG): container finished" podID="8e62eede-ea1b-415e-8549-3df93215459e" containerID="1eed94809c2cc96c8d5e9ea14d6e792498e4a77263e558dd988feb6fa7d9f564" exitCode=0
Sep 18 16:41:13.764541 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:13.764079    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7" event={"ID":"8e62eede-ea1b-415e-8549-3df93215459e","Type":"ContainerDied","Data":"1eed94809c2cc96c8d5e9ea14d6e792498e4a77263e558dd988feb6fa7d9f564"}
Sep 18 16:41:13.764541 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:13.764133    2546 scope.go:117] "RemoveContainer" containerID="3cb91fd86824f74c5df2771e2af0257629209ab209887c473a880e2367c2d2c5"
Sep 18 16:41:13.764541 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:13.764493    2546 scope.go:117] "RemoveContainer" containerID="1eed94809c2cc96c8d5e9ea14d6e792498e4a77263e558dd988feb6fa7d9f564"
Sep 18 16:41:13.765977 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:13.765873    2546 generic.go:356] "Generic (PLEG): container finished" podID="45b59ace-62a2-45ae-9f58-a3d21c8c2ab5" containerID="054d86832b7dfca108df9e1685fffa11e2b3ce1838b0b67ce7c244be56696877" exitCode=0
Sep 18 16:41:13.765977 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:13.765909    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd" event={"ID":"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5","Type":"ContainerDied","Data":"054d86832b7dfca108df9e1685fffa11e2b3ce1838b0b67ce7c244be56696877"}
Sep 18 16:41:13.766250 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:13.766231    2546 scope.go:117] "RemoveContainer" containerID="054d86832b7dfca108df9e1685fffa11e2b3ce1838b0b67ce7c244be56696877"
Sep 18 16:41:14.773199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:14.773159    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7" event={"ID":"8e62eede-ea1b-415e-8549-3df93215459e","Type":"ContainerStarted","Data":"a6074eb5378c656a5a0784460d0dfdbb96ee7712559798e3d8e8abaccd3a274b"}
Sep 18 16:41:14.774878 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:14.774847    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd" event={"ID":"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5","Type":"ContainerStarted","Data":"e7a1dbd5e271c32814fa077a1cd714e2153c852101a4e9d4505ffb3437217e8e"}
Sep 18 16:41:16.764301 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:16.764247    2546 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"4ce26bbe-bd40-4653-a16f-2b0c1c87604b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-09-18T16:38:15Z\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-09-18T16:38:15Z\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"64Mi\\\"},\\\"containerID\\\":\\\"cri-o://082ac3710cb649a00e0130d32f7918b311fcb6280e05340bab5e2d2ec192fe22\\\",\\\"image\\\":\\\"quay.io/opendatahub/odh-model-controller:odh-v3.5-EA2\\\",\\\"imageID\\\":\\\"quay.io/opendatahub/odh-model-controller@sha256:1e2397cbb77c598f2eaff55caa3f0ef39f9f499bc7388d4d92be8d2a5d5ff121\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://536ee85a2a92b0402144cad8b3453f6dd54c4a74f49bcf0a141ee2c7661b28cd\\\",\\\"exitCode\\\":1,\\\"finishedAt\\\":\\\"2026-09-18T16:38:02Z\\\",\\\"reason\\\":\\\"Error\\\",\\\"startedAt\\\":\\\"2026-09-18T16:12:00Z\\\"}},\\\"name\\\":\\\"manager\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"500m\\\",\\\"memory\\\":\\\"2Gi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"10m\\\",\\\"memory\\\":\\\"64Mi\\\"}},\\\"restartCount\\\":1,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-09-18T16:38:04Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000640000],\\\"uid\\\":1000640000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/tmp/k8s-webhook-server/serving-certs\\\",\\\"name\\\":\\\"cert\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-zs6pb\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"opendatahub\"/\"odh-model-controller-7998c67dc4-klwkg\": Timeout: request did not complete within requested timeout - context deadline exceeded" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:41:19.805196 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:19.805163    2546 generic.go:356] "Generic (PLEG): container finished" podID="5d260854-98c0-4dd7-89a4-d9b9609d3203" containerID="4515a05e11c422c5e459cd747ae4a4bb6108fdfed150fc766e5529f7d4054c51" exitCode=1
Sep 18 16:41:19.805678 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:19.805232    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerDied","Data":"4515a05e11c422c5e459cd747ae4a4bb6108fdfed150fc766e5529f7d4054c51"}
Sep 18 16:41:19.805678 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:19.805282    2546 scope.go:117] "RemoveContainer" containerID="71f41230ea0c43f34fec260fa25087faee67f6ac34948b53875ff6673fac4a47"
Sep 18 16:41:19.805799 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:19.805689    2546 scope.go:117] "RemoveContainer" containerID="4515a05e11c422c5e459cd747ae4a4bb6108fdfed150fc766e5529f7d4054c51"
Sep 18 16:41:19.805962 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:19.805938    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:41:21.450652 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:21.450615    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:41:22.934561 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:22.934514    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:41:25.687941 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:25.687814    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{authorino-operator-76b478bd78-t7kmf.18d677ac07f7facb  kuadrant-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kuadrant-system,Name:authorino-operator-76b478bd78-t7kmf,UID:f7e45976-cc30-418c-96f4-70e0c667572e,APIVersion:v1,ResourceVersion:16163,FieldPath:spec.containers{manager},},Reason:Pulled,Message:Container image \"registry.redhat.io/rhcl-1/authorino-rhel9-operator@sha256:8a8cc5fb104032c23096f3c9cf2fbc3636d343ee7ee7974e51fe1f10d3a57589\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:38:02.678123211 +0000 UTC m=+1978.586903561,LastTimestamp:2026-09-18 16:38:02.678123211 +0000 UTC m=+1978.586903561,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:41:34.597316 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:34.597282    2546 scope.go:117] "RemoveContainer" containerID="4515a05e11c422c5e459cd747ae4a4bb6108fdfed150fc766e5529f7d4054c51"
Sep 18 16:41:34.597790 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:34.597463    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:41:39.379976 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:39.379919    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:29Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:29Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:29Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:29Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-135-253.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:41:39.935757 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:39.935707    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s"
Sep 18 16:41:41.796691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:41.796642    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused"
Sep 18 16:41:41.938229 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:41.938193    2546 generic.go:356] "Generic (PLEG): container finished" podID="1c53eeff-c125-49c6-ab26-182dd935f398" containerID="91052c7648f4751de2d6422199ec5006f72203e0c0d6f23b205f796abdc7e95f" exitCode=0
Sep 18 16:41:41.938405 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:41.938263    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" event={"ID":"1c53eeff-c125-49c6-ab26-182dd935f398","Type":"ContainerDied","Data":"91052c7648f4751de2d6422199ec5006f72203e0c0d6f23b205f796abdc7e95f"}
Sep 18 16:41:41.938405 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:41.938329    2546 scope.go:117] "RemoveContainer" containerID="df9864ed5a6bb54db9cd5a5cd3f4d5c4751b13a3865a1f7de0a0cfc58b6b2d0c"
Sep 18 16:41:41.938705 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:41.938683    2546 scope.go:117] "RemoveContainer" containerID="91052c7648f4751de2d6422199ec5006f72203e0c0d6f23b205f796abdc7e95f"
Sep 18 16:41:41.938956 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:41.938933    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-l7p9n_kuadrant-system(1c53eeff-c125-49c6-ab26-182dd935f398)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podUID="1c53eeff-c125-49c6-ab26-182dd935f398"
Sep 18 16:41:43.450516 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:43.450467    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:41:43.450924 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:43.450840    2546 scope.go:117] "RemoveContainer" containerID="91052c7648f4751de2d6422199ec5006f72203e0c0d6f23b205f796abdc7e95f"
Sep 18 16:41:43.451093 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:43.451073    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-l7p9n_kuadrant-system(1c53eeff-c125-49c6-ab26-182dd935f398)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podUID="1c53eeff-c125-49c6-ab26-182dd935f398"
Sep 18 16:41:45.597246 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:45.597206    2546 scope.go:117] "RemoveContainer" containerID="4515a05e11c422c5e459cd747ae4a4bb6108fdfed150fc766e5529f7d4054c51"
Sep 18 16:41:45.962685 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:45.962648    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerStarted","Data":"1b1e796718c7d35132e0097e632d12ac42dc229b5d063e4552daf77503ebe2d6"}
Sep 18 16:41:49.381041 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:49.380977    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:41:49.381041 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:49.381012    2546 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Sep 18 16:41:51.796120 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:51.796064    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:41:51.796578 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:41:51.796561    2546 scope.go:117] "RemoveContainer" containerID="91052c7648f4751de2d6422199ec5006f72203e0c0d6f23b205f796abdc7e95f"
Sep 18 16:41:51.796797 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:51.796776    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-l7p9n_kuadrant-system(1c53eeff-c125-49c6-ab26-182dd935f398)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podUID="1c53eeff-c125-49c6-ab26-182dd935f398"
Sep 18 16:41:56.537543 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:56.537484    2546 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"1c53eeff-c125-49c6-ab26-182dd935f398\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-09-18T16:38:31Z\\\",\\\"type\\\":\\\"Ready\\\"},{\\\"lastTransitionTime\\\":\\\"2026-09-18T16:38:31Z\\\",\\\"type\\\":\\\"ContainersReady\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://91052c7648f4751de2d6422199ec5006f72203e0c0d6f23b205f796abdc7e95f\\\",\\\"image\\\":\\\"registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e\\\",\\\"imageID\\\":\\\"registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:22341c0a543abdcd6cecb58e0e84ffeb713799db22bdf3c01e38cf5c248aee0e\\\",\\\"lastState\\\":{\\\"terminated\\\":{\\\"containerID\\\":\\\"cri-o://df9864ed5a6bb54db9cd5a5cd3f4d5c4751b13a3865a1f7de0a0cfc58b6b2d0c\\\",\\\"exitCode\\\":0,\\\"finishedAt\\\":\\\"2026-09-18T16:38:03Z\\\",\\\"reason\\\":\\\"Completed\\\",\\\"startedAt\\\":\\\"2026-09-18T16:32:43Z\\\"}},\\\"name\\\":\\\"manager\\\",\\\"ready\\\":true,\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"300Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":2,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-09-18T16:38:20Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000680000],\\\"uid\\\":1000680000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/tmp/kuadrant\\\",\\\"name\\\":\\\"extensions-socket-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-xxzwn\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}]}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-5769f99d9f-l7p9n\": etcdserver: request timed out" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:41:56.937982 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:56.937879    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:41:59.694175 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:41:59.694083    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{limitador-operator-controller-manager-74495c69f8-dbsdv.18d676586368f421  kuadrant-system   18829 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kuadrant-system,Name:limitador-operator-controller-manager-74495c69f8-dbsdv,UID:61d67412-30af-4fb0-adc0-5251f88b7a41,APIVersion:v1,ResourceVersion:18696,FieldPath:spec.containers{manager},},Reason:Created,Message:Created container: manager,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:13:43 +0000 UTC,LastTimestamp:2026-09-18 16:38:02.821276119 +0000 UTC m=+1978.730056462,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:42:04.597309 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:04.597268    2546 scope.go:117] "RemoveContainer" containerID="91052c7648f4751de2d6422199ec5006f72203e0c0d6f23b205f796abdc7e95f"
Sep 18 16:42:05.065700 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:05.065662    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" event={"ID":"1c53eeff-c125-49c6-ab26-182dd935f398","Type":"ContainerStarted","Data":"e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843"}
Sep 18 16:42:05.065910 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:05.065891    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:42:09.700591 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:09.700544    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:59Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:59Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:59Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:41:59Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-135-253.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal/status?timeout=10s\": context deadline exceeded"
Sep 18 16:42:10.094417 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:10.094328    2546 generic.go:356] "Generic (PLEG): container finished" podID="61d67412-30af-4fb0-adc0-5251f88b7a41" containerID="c84ba3c21c11d4dc01d7be3d0106b61695f39004e266889e7d87594d1f65f2e0" exitCode=1
Sep 18 16:42:10.094417 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:10.094396    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" event={"ID":"61d67412-30af-4fb0-adc0-5251f88b7a41","Type":"ContainerDied","Data":"c84ba3c21c11d4dc01d7be3d0106b61695f39004e266889e7d87594d1f65f2e0"}
Sep 18 16:42:10.094647 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:10.094477    2546 scope.go:117] "RemoveContainer" containerID="324122dd9d31d30f4c518b31ef077ad5febc35257d54ba486e28086f31a52233"
Sep 18 16:42:10.094838 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:10.094821    2546 scope.go:117] "RemoveContainer" containerID="c84ba3c21c11d4dc01d7be3d0106b61695f39004e266889e7d87594d1f65f2e0"
Sep 18 16:42:10.095154 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:10.095127    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=limitador-operator-controller-manager-74495c69f8-dbsdv_kuadrant-system(61d67412-30af-4fb0-adc0-5251f88b7a41)\"" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" podUID="61d67412-30af-4fb0-adc0-5251f88b7a41"
Sep 18 16:42:10.096723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:10.096702    2546 generic.go:356] "Generic (PLEG): container finished" podID="f7e45976-cc30-418c-96f4-70e0c667572e" containerID="b7334fbd6b0e98847e2e4fb07a38bb00c701eccddfa90666692ae04e16f6f467" exitCode=1
Sep 18 16:42:10.096834 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:10.096770    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" event={"ID":"f7e45976-cc30-418c-96f4-70e0c667572e","Type":"ContainerDied","Data":"b7334fbd6b0e98847e2e4fb07a38bb00c701eccddfa90666692ae04e16f6f467"}
Sep 18 16:42:10.097098 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:10.097084    2546 scope.go:117] "RemoveContainer" containerID="b7334fbd6b0e98847e2e4fb07a38bb00c701eccddfa90666692ae04e16f6f467"
Sep 18 16:42:10.097287 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:10.097270    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=authorino-operator-76b478bd78-t7kmf_kuadrant-system(f7e45976-cc30-418c-96f4-70e0c667572e)\"" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" podUID="f7e45976-cc30-418c-96f4-70e0c667572e"
Sep 18 16:42:10.105792 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:10.105774    2546 scope.go:117] "RemoveContainer" containerID="c9e1290130d2c65bb0fe3bcbf5b1c61eec051bc0cb3273a15f51a611fb97a9ff"
Sep 18 16:42:13.939587 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:13.939524    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:42:14.143481 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:14.143449    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:42:14.143803 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:14.143789    2546 scope.go:117] "RemoveContainer" containerID="b7334fbd6b0e98847e2e4fb07a38bb00c701eccddfa90666692ae04e16f6f467"
Sep 18 16:42:14.144067 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:14.144014    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=authorino-operator-76b478bd78-t7kmf_kuadrant-system(f7e45976-cc30-418c-96f4-70e0c667572e)\"" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" podUID="f7e45976-cc30-418c-96f4-70e0c667572e"
Sep 18 16:42:14.701145 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:14.701090    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:42:14.701354 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:14.701174    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:42:14.701484 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:14.701471    2546 scope.go:117] "RemoveContainer" containerID="c84ba3c21c11d4dc01d7be3d0106b61695f39004e266889e7d87594d1f65f2e0"
Sep 18 16:42:14.701697 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:14.701677    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=limitador-operator-controller-manager-74495c69f8-dbsdv_kuadrant-system(61d67412-30af-4fb0-adc0-5251f88b7a41)\"" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" podUID="61d67412-30af-4fb0-adc0-5251f88b7a41"
Sep 18 16:42:15.127641 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:15.127557    2546 scope.go:117] "RemoveContainer" containerID="b7334fbd6b0e98847e2e4fb07a38bb00c701eccddfa90666692ae04e16f6f467"
Sep 18 16:42:15.128003 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:15.127755    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=authorino-operator-76b478bd78-t7kmf_kuadrant-system(f7e45976-cc30-418c-96f4-70e0c667572e)\"" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" podUID="f7e45976-cc30-418c-96f4-70e0c667572e"
Sep 18 16:42:16.073197 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:16.073161    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:42:19.701553 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:19.701509    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:42:23.520372 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:23.520329    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:42:23.520786 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:23.520721    2546 scope.go:117] "RemoveContainer" containerID="c84ba3c21c11d4dc01d7be3d0106b61695f39004e266889e7d87594d1f65f2e0"
Sep 18 16:42:24.176089 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:24.176043    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" event={"ID":"61d67412-30af-4fb0-adc0-5251f88b7a41","Type":"ContainerStarted","Data":"e45640e47570f5a8708515d9979875695e9da23496b279081c84459551368b55"}
Sep 18 16:42:24.176326 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:24.176305    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:42:26.596545 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:26.596512    2546 scope.go:117] "RemoveContainer" containerID="b7334fbd6b0e98847e2e4fb07a38bb00c701eccddfa90666692ae04e16f6f467"
Sep 18 16:42:27.196349 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:27.196315    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" event={"ID":"f7e45976-cc30-418c-96f4-70e0c667572e","Type":"ContainerStarted","Data":"f85beff468102e90e392b320d6751adbc123af6c350360ab8f43c9bc0cd0192b"}
Sep 18 16:42:27.196544 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:27.196524    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:42:29.702560 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:29.702518    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:42:30.940721 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:30.940676    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:42:33.697606 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:33.697512    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{authorino-operator-76b478bd78-t7kmf.18d67652235f91df  kuadrant-system   16712 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kuadrant-system,Name:authorino-operator-76b478bd78-t7kmf,UID:f7e45976-cc30-418c-96f4-70e0c667572e,APIVersion:v1,ResourceVersion:16163,FieldPath:spec.containers{manager},},Reason:Created,Message:Created container: manager,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:13:17 +0000 UTC,LastTimestamp:2026-09-18 16:38:02.822241493 +0000 UTC m=+1978.731021845,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:42:35.183984 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:35.183952    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:42:38.204000 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:38.203964    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:42:39.702936 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:39.702900    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:42:46.296575 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:46.296537    2546 generic.go:356] "Generic (PLEG): container finished" podID="5d260854-98c0-4dd7-89a4-d9b9609d3203" containerID="1b1e796718c7d35132e0097e632d12ac42dc229b5d063e4552daf77503ebe2d6" exitCode=1
Sep 18 16:42:46.297102 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:46.296607    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerDied","Data":"1b1e796718c7d35132e0097e632d12ac42dc229b5d063e4552daf77503ebe2d6"}
Sep 18 16:42:46.297102 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:46.296654    2546 scope.go:117] "RemoveContainer" containerID="4515a05e11c422c5e459cd747ae4a4bb6108fdfed150fc766e5529f7d4054c51"
Sep 18 16:42:46.297102 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:42:46.297006    2546 scope.go:117] "RemoveContainer" containerID="1b1e796718c7d35132e0097e632d12ac42dc229b5d063e4552daf77503ebe2d6"
Sep 18 16:42:46.297325 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:46.297296    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:42:47.942574 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:47.942533    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:42:49.703728 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:49.703691    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:42:49.703728 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:49.703721    2546 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Sep 18 16:42:57.618043 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:42:57.617985    2546 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods kuadrant-operator-controller-manager-5769f99d9f-l7p9n)" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:43:00.378141 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:00.378110    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57cdc4b768-7sgr7_8e62eede-ea1b-415e-8549-3df93215459e/service-ca-operator/2.log"
Sep 18 16:43:00.378561 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:00.378475    2546 generic.go:356] "Generic (PLEG): container finished" podID="8e62eede-ea1b-415e-8549-3df93215459e" containerID="a6074eb5378c656a5a0784460d0dfdbb96ee7712559798e3d8e8abaccd3a274b" exitCode=255
Sep 18 16:43:00.378703 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:00.378555    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7" event={"ID":"8e62eede-ea1b-415e-8549-3df93215459e","Type":"ContainerDied","Data":"a6074eb5378c656a5a0784460d0dfdbb96ee7712559798e3d8e8abaccd3a274b"}
Sep 18 16:43:00.378703 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:00.378621    2546 scope.go:117] "RemoveContainer" containerID="1eed94809c2cc96c8d5e9ea14d6e792498e4a77263e558dd988feb6fa7d9f564"
Sep 18 16:43:00.378979 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:00.378966    2546 scope.go:117] "RemoveContainer" containerID="a6074eb5378c656a5a0784460d0dfdbb96ee7712559798e3d8e8abaccd3a274b"
Sep 18 16:43:00.379249 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:43:00.379225    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"service-ca-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=service-ca-operator pod=service-ca-operator-57cdc4b768-7sgr7_openshift-service-ca-operator(8e62eede-ea1b-415e-8549-3df93215459e)\"" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7" podUID="8e62eede-ea1b-415e-8549-3df93215459e"
Sep 18 16:43:00.380368 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:00.380348    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-55f6fc7b86-7xldd_45b59ace-62a2-45ae-9f58-a3d21c8c2ab5/service-ca-controller/1.log"
Sep 18 16:43:00.380761 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:00.380739    2546 generic.go:356] "Generic (PLEG): container finished" podID="45b59ace-62a2-45ae-9f58-a3d21c8c2ab5" containerID="e7a1dbd5e271c32814fa077a1cd714e2153c852101a4e9d4505ffb3437217e8e" exitCode=255
Sep 18 16:43:00.380857 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:00.380809    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd" event={"ID":"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5","Type":"ContainerDied","Data":"e7a1dbd5e271c32814fa077a1cd714e2153c852101a4e9d4505ffb3437217e8e"}
Sep 18 16:43:00.381128 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:00.381109    2546 scope.go:117] "RemoveContainer" containerID="e7a1dbd5e271c32814fa077a1cd714e2153c852101a4e9d4505ffb3437217e8e"
Sep 18 16:43:00.381309 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:43:00.381293    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"service-ca-controller\" with CrashLoopBackOff: \"back-off 10s restarting failed container=service-ca-controller pod=service-ca-55f6fc7b86-7xldd_openshift-service-ca(45b59ace-62a2-45ae-9f58-a3d21c8c2ab5)\"" pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd" podUID="45b59ace-62a2-45ae-9f58-a3d21c8c2ab5"
Sep 18 16:43:00.390129 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:00.390107    2546 scope.go:117] "RemoveContainer" containerID="054d86832b7dfca108df9e1685fffa11e2b3ce1838b0b67ce7c244be56696877"
Sep 18 16:43:01.387716 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:01.387688    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57cdc4b768-7sgr7_8e62eede-ea1b-415e-8549-3df93215459e/service-ca-operator/2.log"
Sep 18 16:43:01.389351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:01.389329    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-55f6fc7b86-7xldd_45b59ace-62a2-45ae-9f58-a3d21c8c2ab5/service-ca-controller/1.log"
Sep 18 16:43:01.597172 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:01.597142    2546 scope.go:117] "RemoveContainer" containerID="1b1e796718c7d35132e0097e632d12ac42dc229b5d063e4552daf77503ebe2d6"
Sep 18 16:43:01.597402 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:43:01.597368    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:43:04.943815 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:43:04.943765    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s"
Sep 18 16:43:07.701999 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:43:07.701879    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{limitador-operator-controller-manager-74495c69f8-dbsdv.18d67658647aee60  kuadrant-system   18834 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kuadrant-system,Name:limitador-operator-controller-manager-74495c69f8-dbsdv,UID:61d67412-30af-4fb0-adc0-5251f88b7a41,APIVersion:v1,ResourceVersion:18696,FieldPath:spec.containers{manager},},Reason:Started,Message:Started container manager,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:13:43 +0000 UTC,LastTimestamp:2026-09-18 16:38:02.829482222 +0000 UTC m=+1978.738262554,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:43:13.596920 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:13.596872    2546 scope.go:117] "RemoveContainer" containerID="1b1e796718c7d35132e0097e632d12ac42dc229b5d063e4552daf77503ebe2d6"
Sep 18 16:43:13.597376 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:43:13.597100    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:43:14.597396 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:14.597363    2546 scope.go:117] "RemoveContainer" containerID="e7a1dbd5e271c32814fa077a1cd714e2153c852101a4e9d4505ffb3437217e8e"
Sep 18 16:43:15.462315 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:15.462283    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-55f6fc7b86-7xldd_45b59ace-62a2-45ae-9f58-a3d21c8c2ab5/service-ca-controller/1.log"
Sep 18 16:43:15.462531 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:15.462414    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-7xldd" event={"ID":"45b59ace-62a2-45ae-9f58-a3d21c8c2ab5","Type":"ContainerStarted","Data":"1b560f4e7f88937c5ac79718b0783ab2f8f5ffd2b61d866265f8f49b8297320e"}
Sep 18 16:43:15.597552 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:15.597519    2546 scope.go:117] "RemoveContainer" containerID="a6074eb5378c656a5a0784460d0dfdbb96ee7712559798e3d8e8abaccd3a274b"
Sep 18 16:43:16.469422 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:16.469390    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57cdc4b768-7sgr7_8e62eede-ea1b-415e-8549-3df93215459e/service-ca-operator/2.log"
Sep 18 16:43:16.469618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:16.469515    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-7sgr7" event={"ID":"8e62eede-ea1b-415e-8549-3df93215459e","Type":"ContainerStarted","Data":"e8de7fe4375cc626a14c25c7d0b2928ebffa332d68c1fb0202a57aed32313757"}
Sep 18 16:43:21.944858 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:43:21.944816    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s"
Sep 18 16:43:24.597445 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:24.597405    2546 scope.go:117] "RemoveContainer" containerID="1b1e796718c7d35132e0097e632d12ac42dc229b5d063e4552daf77503ebe2d6"
Sep 18 16:43:24.597852 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:43:24.597657    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:43:38.596827 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:38.596794    2546 scope.go:117] "RemoveContainer" containerID="1b1e796718c7d35132e0097e632d12ac42dc229b5d063e4552daf77503ebe2d6"
Sep 18 16:43:38.946079 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:43:38.945997    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:43:39.594729 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:43:39.594691    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerStarted","Data":"debfef9f6428b8d3ffa9087097c6e36270ed679f4872714ba8b78faf3641950a"}
Sep 18 16:43:40.457508 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:43:40.457457    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:43:30Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:43:30Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:43:30Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:43:30Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-135-253.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:43:41.705853 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:43:41.705755    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{authorino-operator-76b478bd78-t7kmf.18d6765223df989c  kuadrant-system   16715 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kuadrant-system,Name:authorino-operator-76b478bd78-t7kmf,UID:f7e45976-cc30-418c-96f4-70e0c667572e,APIVersion:v1,ResourceVersion:16163,FieldPath:spec.containers{manager},},Reason:Started,Message:Started container manager,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:13:17 +0000 UTC,LastTimestamp:2026-09-18 16:38:02.830268725 +0000 UTC m=+1978.739049075,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:43:50.457840 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:43:50.457796    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:44:13.691185 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:13.691137    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" podUID="d9b87cf9-1252-4eeb-bd58-194c81ea4a94" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/readyz\": dial tcp 10.133.0.27:8081: connect: connection refused"
Sep 18 16:44:13.780140 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:13.780105    2546 generic.go:356] "Generic (PLEG): container finished" podID="d9b87cf9-1252-4eeb-bd58-194c81ea4a94" containerID="7ffa2bf0a0351755ec4c7d243d09d324b5c16d217e2986a9043f161fd394f65d" exitCode=1
Sep 18 16:44:13.780311 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:13.780179    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" event={"ID":"d9b87cf9-1252-4eeb-bd58-194c81ea4a94","Type":"ContainerDied","Data":"7ffa2bf0a0351755ec4c7d243d09d324b5c16d217e2986a9043f161fd394f65d"}
Sep 18 16:44:13.780311 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:13.780244    2546 scope.go:117] "RemoveContainer" containerID="38dcdd6f151f25ff14e35fca9d69d099ae915f1b50af9d604fb1bb1917144906"
Sep 18 16:44:13.780643 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:13.780613    2546 scope.go:117] "RemoveContainer" containerID="7ffa2bf0a0351755ec4c7d243d09d324b5c16d217e2986a9043f161fd394f65d"
Sep 18 16:44:13.780894 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:13.780873    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-8bfgv_opendatahub(d9b87cf9-1252-4eeb-bd58-194c81ea4a94)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" podUID="d9b87cf9-1252-4eeb-bd58-194c81ea4a94"
Sep 18 16:44:13.782294 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:13.782272    2546 generic.go:356] "Generic (PLEG): container finished" podID="4ce26bbe-bd40-4653-a16f-2b0c1c87604b" containerID="082ac3710cb649a00e0130d32f7918b311fcb6280e05340bab5e2d2ec192fe22" exitCode=1
Sep 18 16:44:13.782346 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:13.782317    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" event={"ID":"4ce26bbe-bd40-4653-a16f-2b0c1c87604b","Type":"ContainerDied","Data":"082ac3710cb649a00e0130d32f7918b311fcb6280e05340bab5e2d2ec192fe22"}
Sep 18 16:44:13.782682 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:13.782663    2546 scope.go:117] "RemoveContainer" containerID="082ac3710cb649a00e0130d32f7918b311fcb6280e05340bab5e2d2ec192fe22"
Sep 18 16:44:13.782907 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:13.782890    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-model-controller-7998c67dc4-klwkg_opendatahub(4ce26bbe-bd40-4653-a16f-2b0c1c87604b)\"" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" podUID="4ce26bbe-bd40-4653-a16f-2b0c1c87604b"
Sep 18 16:44:13.792770 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:13.792751    2546 scope.go:117] "RemoveContainer" containerID="536ee85a2a92b0402144cad8b3453f6dd54c4a74f49bcf0a141ee2c7661b28cd"
Sep 18 16:44:14.727819 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:14.727782    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:44:14.753950 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:14.753922    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:44:14.789839 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:14.789816    2546 scope.go:117] "RemoveContainer" containerID="7ffa2bf0a0351755ec4c7d243d09d324b5c16d217e2986a9043f161fd394f65d"
Sep 18 16:44:14.790122 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:14.790102    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-8bfgv_opendatahub(d9b87cf9-1252-4eeb-bd58-194c81ea4a94)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" podUID="d9b87cf9-1252-4eeb-bd58-194c81ea4a94"
Sep 18 16:44:14.791467 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:14.791447    2546 generic.go:356] "Generic (PLEG): container finished" podID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerID="ec7a0640f0844f3a40d039c2f480dfcb228250877b0dee0820cd1c757acd9233" exitCode=1
Sep 18 16:44:14.791569 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:14.791514    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerDied","Data":"ec7a0640f0844f3a40d039c2f480dfcb228250877b0dee0820cd1c757acd9233"}
Sep 18 16:44:14.791616 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:14.791571    2546 scope.go:117] "RemoveContainer" containerID="778bb1ef5585ff103e7dc27354d8781cc8f5383595cafc9fccc126ca3c7970dd"
Sep 18 16:44:14.791938 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:14.791922    2546 scope.go:117] "RemoveContainer" containerID="ec7a0640f0844f3a40d039c2f480dfcb228250877b0dee0820cd1c757acd9233"
Sep 18 16:44:14.792210 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:14.792185    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:44:14.793523 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:14.793506    2546 scope.go:117] "RemoveContainer" containerID="082ac3710cb649a00e0130d32f7918b311fcb6280e05340bab5e2d2ec192fe22"
Sep 18 16:44:14.793666 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:14.793649    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-model-controller-7998c67dc4-klwkg_opendatahub(4ce26bbe-bd40-4653-a16f-2b0c1c87604b)\"" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" podUID="4ce26bbe-bd40-4653-a16f-2b0c1c87604b"
Sep 18 16:44:15.705356 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:15.705320    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:44:15.705548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:15.705424    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:44:15.710324 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:15.710241    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{kuadrant-operator-controller-manager-5769f99d9f-l7p9n.18d67761b48d696e  kuadrant-system   49952 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kuadrant-system,Name:kuadrant-operator-controller-manager-5769f99d9f-l7p9n,UID:1c53eeff-c125-49c6-ab26-182dd935f398,APIVersion:v1,ResourceVersion:18673,FieldPath:spec.containers{manager},},Reason:Unhealthy,Message:Liveness probe failed: Get \"http://10.133.0.34:8081/healthz\": dial tcp 10.133.0.34:8081: connect: connection refused,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:32:43 +0000 UTC,LastTimestamp:2026-09-18 16:38:03.451306038 +0000 UTC m=+1979.360086391,Count:2,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:44:15.800247 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:15.800218    2546 scope.go:117] "RemoveContainer" containerID="ec7a0640f0844f3a40d039c2f480dfcb228250877b0dee0820cd1c757acd9233"
Sep 18 16:44:15.800619 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:15.800282    2546 scope.go:117] "RemoveContainer" containerID="082ac3710cb649a00e0130d32f7918b311fcb6280e05340bab5e2d2ec192fe22"
Sep 18 16:44:15.800619 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:15.800414    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:44:15.800619 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:15.800431    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-model-controller-7998c67dc4-klwkg_opendatahub(4ce26bbe-bd40-4653-a16f-2b0c1c87604b)\"" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" podUID="4ce26bbe-bd40-4653-a16f-2b0c1c87604b"
Sep 18 16:44:18.690212 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:18.690170    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:44:18.690625 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:18.690594    2546 scope.go:117] "RemoveContainer" containerID="7ffa2bf0a0351755ec4c7d243d09d324b5c16d217e2986a9043f161fd394f65d"
Sep 18 16:44:18.690818 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:18.690799    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-8bfgv_opendatahub(d9b87cf9-1252-4eeb-bd58-194c81ea4a94)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" podUID="d9b87cf9-1252-4eeb-bd58-194c81ea4a94"
Sep 18 16:44:20.717702 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:20.717660    2546 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:44:24.409624 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:24.409580    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:44:24.410172 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:24.410097    2546 scope.go:117] "RemoveContainer" containerID="ec7a0640f0844f3a40d039c2f480dfcb228250877b0dee0820cd1c757acd9233"
Sep 18 16:44:24.410367 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:24.410342    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:44:29.597185 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:29.597149    2546 scope.go:117] "RemoveContainer" containerID="082ac3710cb649a00e0130d32f7918b311fcb6280e05340bab5e2d2ec192fe22"
Sep 18 16:44:29.597585 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:29.597453    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:44:30.718384 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:30.718342    2546 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:44:30.877539 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:30.877433    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" event={"ID":"4ce26bbe-bd40-4653-a16f-2b0c1c87604b","Type":"ContainerStarted","Data":"45ddde9ad92050db8a056a7194d049e3904b4e4ecee255bf5af368158417eea1"}
Sep 18 16:44:30.877727 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:30.877691    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:44:32.597566 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:32.597526    2546 scope.go:117] "RemoveContainer" containerID="7ffa2bf0a0351755ec4c7d243d09d324b5c16d217e2986a9043f161fd394f65d"
Sep 18 16:44:33.008779 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.008745    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"]
Sep 18 16:44:33.009149 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.009136    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249" containerName="test-curl-7582-10b845"
Sep 18 16:44:33.009199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.009151    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249" containerName="test-curl-7582-10b845"
Sep 18 16:44:33.009199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.009172    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de443cc3-cf0a-452c-969f-a404401d7e8f" containerName="main"
Sep 18 16:44:33.009199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.009177    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="de443cc3-cf0a-452c-969f-a404401d7e8f" containerName="main"
Sep 18 16:44:33.009301 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.009210    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47053e5a-808a-4d56-9b56-9524941908b0" containerName="main"
Sep 18 16:44:33.009301 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.009216    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="47053e5a-808a-4d56-9b56-9524941908b0" containerName="main"
Sep 18 16:44:33.009301 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.009276    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="47053e5a-808a-4d56-9b56-9524941908b0" containerName="main"
Sep 18 16:44:33.009301 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.009285    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d51ed9b-5a7d-4f63-82ea-bcf6e67cb249" containerName="test-curl-7582-10b845"
Sep 18 16:44:33.009301 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.009292    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="de443cc3-cf0a-452c-969f-a404401d7e8f" containerName="main"
Sep 18 16:44:33.012394 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.012375    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.015689 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.015670    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"default-dockercfg-jshkt\""
Sep 18 16:44:33.016712 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.016687    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"openshift-service-ca.crt\""
Sep 18 16:44:33.016827 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.016707    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"rate-test-model-a75117-kserve-self-signed-certs\""
Sep 18 16:44:33.016827 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:33.016690    2546 controller.go:195] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"ip-10-0-135-253.ec2.internal\": the object has been modified; please apply your changes to the latest version and try again"
Sep 18 16:44:33.018133 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.018114    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"kube-root-ca.crt\""
Sep 18 16:44:33.034733 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.034704    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"]
Sep 18 16:44:33.132418 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.132376    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a6b782-55b5-4c60-86fd-cf41221abc99-tmp-dir\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.132620 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.132445    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a6b782-55b5-4c60-86fd-cf41221abc99-tls-certs\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.132620 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.132484    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a6b782-55b5-4c60-86fd-cf41221abc99-dshm\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.132620 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.132521    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a6b782-55b5-4c60-86fd-cf41221abc99-model-cache\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.132620 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.132550    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnd6w\" (UniqueName: \"kubernetes.io/projected/60a6b782-55b5-4c60-86fd-cf41221abc99-kube-api-access-wnd6w\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.132790 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.132630    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a6b782-55b5-4c60-86fd-cf41221abc99-home\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.233779 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.233738    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnd6w\" (UniqueName: \"kubernetes.io/projected/60a6b782-55b5-4c60-86fd-cf41221abc99-kube-api-access-wnd6w\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.233997 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.233799    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a6b782-55b5-4c60-86fd-cf41221abc99-home\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.233997 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.233971    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a6b782-55b5-4c60-86fd-cf41221abc99-tmp-dir\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.234182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.234087    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a6b782-55b5-4c60-86fd-cf41221abc99-tls-certs\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.234182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.234133    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a6b782-55b5-4c60-86fd-cf41221abc99-dshm\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.234281 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.234183    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a6b782-55b5-4c60-86fd-cf41221abc99-model-cache\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.234281 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.234210    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60a6b782-55b5-4c60-86fd-cf41221abc99-home\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.234355 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.234288    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60a6b782-55b5-4c60-86fd-cf41221abc99-tmp-dir\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.234473 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.234456    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60a6b782-55b5-4c60-86fd-cf41221abc99-model-cache\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.236364 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.236326    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60a6b782-55b5-4c60-86fd-cf41221abc99-dshm\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.236725 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.236709    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60a6b782-55b5-4c60-86fd-cf41221abc99-tls-certs\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.249447 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.249415    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnd6w\" (UniqueName: \"kubernetes.io/projected/60a6b782-55b5-4c60-86fd-cf41221abc99-kube-api-access-wnd6w\") pod \"rate-test-model-a75117-kserve-8668d4d96c-xz6xx\" (UID: \"60a6b782-55b5-4c60-86fd-cf41221abc99\") " pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.321714 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.321681    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:33.603390 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.603363    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"]
Sep 18 16:44:33.604931 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:44:33.604896    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60a6b782_55b5_4c60_86fd_cf41221abc99.slice/crio-417a1a1be87d79f784bd2c634850560e8d5617049ca5d8392676e56244397b41 WatchSource:0}: Error finding container 417a1a1be87d79f784bd2c634850560e8d5617049ca5d8392676e56244397b41: Status 404 returned error can't find the container with id 417a1a1be87d79f784bd2c634850560e8d5617049ca5d8392676e56244397b41
Sep 18 16:44:33.895406 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.895296    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx" event={"ID":"60a6b782-55b5-4c60-86fd-cf41221abc99","Type":"ContainerStarted","Data":"d0272b7ef85678e71c68210dfcac5a6af42d0f9a8d56200d6db5c8068adab37d"}
Sep 18 16:44:33.895406 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.895353    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx" event={"ID":"60a6b782-55b5-4c60-86fd-cf41221abc99","Type":"ContainerStarted","Data":"417a1a1be87d79f784bd2c634850560e8d5617049ca5d8392676e56244397b41"}
Sep 18 16:44:33.897236 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.897203    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" event={"ID":"d9b87cf9-1252-4eeb-bd58-194c81ea4a94","Type":"ContainerStarted","Data":"91610200fb7e250c873f96d6871f65b3d42e77ca26047d40e3a7cd89ba1f36cd"}
Sep 18 16:44:33.897429 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.897410    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:44:33.919076 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:33.919004    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx" podStartSLOduration=440.918988372 podStartE2EDuration="7m20.918988372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:44:33.916798649 +0000 UTC m=+2369.825579020" watchObservedRunningTime="2026-09-18 16:44:33.918988372 +0000 UTC m=+2369.827768723"
Sep 18 16:44:37.596766 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:37.596721    2546 scope.go:117] "RemoveContainer" containerID="ec7a0640f0844f3a40d039c2f480dfcb228250877b0dee0820cd1c757acd9233"
Sep 18 16:44:38.924923 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:38.924878    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerStarted","Data":"cffca3674c4a9164d4ec101c40d11c66e27300f1642a2cb932176a75d040efea"}
Sep 18 16:44:38.925418 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:38.925186    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:44:38.927001 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:38.926978    2546 generic.go:356] "Generic (PLEG): container finished" podID="5d260854-98c0-4dd7-89a4-d9b9609d3203" containerID="debfef9f6428b8d3ffa9087097c6e36270ed679f4872714ba8b78faf3641950a" exitCode=1
Sep 18 16:44:38.927134 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:38.927046    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerDied","Data":"debfef9f6428b8d3ffa9087097c6e36270ed679f4872714ba8b78faf3641950a"}
Sep 18 16:44:38.927134 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:38.927081    2546 scope.go:117] "RemoveContainer" containerID="1b1e796718c7d35132e0097e632d12ac42dc229b5d063e4552daf77503ebe2d6"
Sep 18 16:44:38.927355 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:38.927339    2546 scope.go:117] "RemoveContainer" containerID="debfef9f6428b8d3ffa9087097c6e36270ed679f4872714ba8b78faf3641950a"
Sep 18 16:44:38.927558 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:38.927537    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:44:41.884882 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:41.884850    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:44:43.322523 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:43.322485    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:43.322975 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:43.322757    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:43.335401 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:43.335376    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:43.974742 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:43.974707    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-a75117/rate-test-model-a75117-kserve-8668d4d96c-xz6xx"
Sep 18 16:44:49.714210 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:49.714106    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{odh-model-controller-7998c67dc4-klwkg.18d6763f07e39a24  opendatahub   14327 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:odh-model-controller-7998c67dc4-klwkg,UID:4ce26bbe-bd40-4653-a16f-2b0c1c87604b,APIVersion:v1,ResourceVersion:14179,FieldPath:spec.containers{manager},},Reason:Pulling,Message:Pulling image \"quay.io/opendatahub/odh-model-controller:odh-v3.5-EA2\",Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:11:55 +0000 UTC,LastTimestamp:2026-09-18 16:38:03.681825046 +0000 UTC m=+1979.590605377,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:44:51.596822 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:44:51.596785    2546 scope.go:117] "RemoveContainer" containerID="debfef9f6428b8d3ffa9087097c6e36270ed679f4872714ba8b78faf3641950a"
Sep 18 16:44:51.597336 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:44:51.597016    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:45:04.599000 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:04.598964    2546 scope.go:117] "RemoveContainer" containerID="debfef9f6428b8d3ffa9087097c6e36270ed679f4872714ba8b78faf3641950a"
Sep 18 16:45:04.599623 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:04.599178    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:45:04.904058 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:04.903952    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:45:05.064751 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:05.064719    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-55f6fc7b86-7xldd_45b59ace-62a2-45ae-9f58-a3d21c8c2ab5/service-ca-controller/1.log"
Sep 18 16:45:05.065537 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:05.065512    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57cdc4b768-7sgr7_8e62eede-ea1b-415e-8549-3df93215459e/service-ca-operator/2.log"
Sep 18 16:45:05.075347 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:05.075323    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-55f6fc7b86-7xldd_45b59ace-62a2-45ae-9f58-a3d21c8c2ab5/service-ca-controller/1.log"
Sep 18 16:45:05.075964 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:05.075949    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57cdc4b768-7sgr7_8e62eede-ea1b-415e-8549-3df93215459e/service-ca-operator/2.log"
Sep 18 16:45:07.093957 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:07.093920    2546 generic.go:356] "Generic (PLEG): container finished" podID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerID="c28e358e7730fc0f82ecd3c76dbfd56ee4f554326adfdc0446501c801d293b3a" exitCode=1
Sep 18 16:45:07.094411 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:07.093983    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerDied","Data":"c28e358e7730fc0f82ecd3c76dbfd56ee4f554326adfdc0446501c801d293b3a"}
Sep 18 16:45:07.094411 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:07.094385    2546 scope.go:117] "RemoveContainer" containerID="c28e358e7730fc0f82ecd3c76dbfd56ee4f554326adfdc0446501c801d293b3a"
Sep 18 16:45:07.096063 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:07.096019    2546 generic.go:356] "Generic (PLEG): container finished" podID="4ce26bbe-bd40-4653-a16f-2b0c1c87604b" containerID="45ddde9ad92050db8a056a7194d049e3904b4e4ecee255bf5af368158417eea1" exitCode=1
Sep 18 16:45:07.096179 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:07.096091    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" event={"ID":"4ce26bbe-bd40-4653-a16f-2b0c1c87604b","Type":"ContainerDied","Data":"45ddde9ad92050db8a056a7194d049e3904b4e4ecee255bf5af368158417eea1"}
Sep 18 16:45:07.096179 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:07.096139    2546 scope.go:117] "RemoveContainer" containerID="082ac3710cb649a00e0130d32f7918b311fcb6280e05340bab5e2d2ec192fe22"
Sep 18 16:45:07.096515 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:07.096495    2546 scope.go:117] "RemoveContainer" containerID="45ddde9ad92050db8a056a7194d049e3904b4e4ecee255bf5af368158417eea1"
Sep 18 16:45:07.096772 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:07.096748    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=odh-model-controller-7998c67dc4-klwkg_opendatahub(4ce26bbe-bd40-4653-a16f-2b0c1c87604b)\"" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" podUID="4ce26bbe-bd40-4653-a16f-2b0c1c87604b"
Sep 18 16:45:07.098314 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:07.098294    2546 generic.go:356] "Generic (PLEG): container finished" podID="1c53eeff-c125-49c6-ab26-182dd935f398" containerID="e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843" exitCode=0
Sep 18 16:45:07.098419 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:07.098363    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" event={"ID":"1c53eeff-c125-49c6-ab26-182dd935f398","Type":"ContainerDied","Data":"e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843"}
Sep 18 16:45:07.098659 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:07.098644    2546 scope.go:117] "RemoveContainer" containerID="e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843"
Sep 18 16:45:07.098844 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:07.098827    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-l7p9n_kuadrant-system(1c53eeff-c125-49c6-ab26-182dd935f398)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podUID="1c53eeff-c125-49c6-ab26-182dd935f398"
Sep 18 16:45:07.107610 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:07.107593    2546 scope.go:117] "RemoveContainer" containerID="91052c7648f4751de2d6422199ec5006f72203e0c0d6f23b205f796abdc7e95f"
Sep 18 16:45:08.106247 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.106213    2546 generic.go:356] "Generic (PLEG): container finished" podID="f7e45976-cc30-418c-96f4-70e0c667572e" containerID="f85beff468102e90e392b320d6751adbc123af6c350360ab8f43c9bc0cd0192b" exitCode=1
Sep 18 16:45:08.106692 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.106284    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" event={"ID":"f7e45976-cc30-418c-96f4-70e0c667572e","Type":"ContainerDied","Data":"f85beff468102e90e392b320d6751adbc123af6c350360ab8f43c9bc0cd0192b"}
Sep 18 16:45:08.106692 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.106345    2546 scope.go:117] "RemoveContainer" containerID="b7334fbd6b0e98847e2e4fb07a38bb00c701eccddfa90666692ae04e16f6f467"
Sep 18 16:45:08.106800 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.106702    2546 scope.go:117] "RemoveContainer" containerID="f85beff468102e90e392b320d6751adbc123af6c350360ab8f43c9bc0cd0192b"
Sep 18 16:45:08.106942 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:08.106913    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=authorino-operator-76b478bd78-t7kmf_kuadrant-system(f7e45976-cc30-418c-96f4-70e0c667572e)\"" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" podUID="f7e45976-cc30-418c-96f4-70e0c667572e"
Sep 18 16:45:08.108985 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.108963    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerStarted","Data":"d2e021c90cb80608a2865e381d816aa43dbfbadf33234b051cd216c9f9c5114e"}
Sep 18 16:45:08.109300 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.109279    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:45:08.114398 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.114377    2546 generic.go:356] "Generic (PLEG): container finished" podID="61d67412-30af-4fb0-adc0-5251f88b7a41" containerID="e45640e47570f5a8708515d9979875695e9da23496b279081c84459551368b55" exitCode=1
Sep 18 16:45:08.114497 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.114441    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" event={"ID":"61d67412-30af-4fb0-adc0-5251f88b7a41","Type":"ContainerDied","Data":"e45640e47570f5a8708515d9979875695e9da23496b279081c84459551368b55"}
Sep 18 16:45:08.114824 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.114807    2546 scope.go:117] "RemoveContainer" containerID="e45640e47570f5a8708515d9979875695e9da23496b279081c84459551368b55"
Sep 18 16:45:08.115050 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:08.115016    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=limitador-operator-controller-manager-74495c69f8-dbsdv_kuadrant-system(61d67412-30af-4fb0-adc0-5251f88b7a41)\"" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" podUID="61d67412-30af-4fb0-adc0-5251f88b7a41"
Sep 18 16:45:08.116356 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.116335    2546 generic.go:356] "Generic (PLEG): container finished" podID="d9b87cf9-1252-4eeb-bd58-194c81ea4a94" containerID="91610200fb7e250c873f96d6871f65b3d42e77ca26047d40e3a7cd89ba1f36cd" exitCode=1
Sep 18 16:45:08.116456 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.116396    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" event={"ID":"d9b87cf9-1252-4eeb-bd58-194c81ea4a94","Type":"ContainerDied","Data":"91610200fb7e250c873f96d6871f65b3d42e77ca26047d40e3a7cd89ba1f36cd"}
Sep 18 16:45:08.116688 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.116670    2546 scope.go:117] "RemoveContainer" containerID="91610200fb7e250c873f96d6871f65b3d42e77ca26047d40e3a7cd89ba1f36cd"
Sep 18 16:45:08.116895 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:08.116879    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-8bfgv_opendatahub(d9b87cf9-1252-4eeb-bd58-194c81ea4a94)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" podUID="d9b87cf9-1252-4eeb-bd58-194c81ea4a94"
Sep 18 16:45:08.118220 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.118203    2546 generic.go:356] "Generic (PLEG): container finished" podID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerID="cffca3674c4a9164d4ec101c40d11c66e27300f1642a2cb932176a75d040efea" exitCode=1
Sep 18 16:45:08.118300 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.118228    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerDied","Data":"cffca3674c4a9164d4ec101c40d11c66e27300f1642a2cb932176a75d040efea"}
Sep 18 16:45:08.118515 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.118502    2546 scope.go:117] "RemoveContainer" containerID="cffca3674c4a9164d4ec101c40d11c66e27300f1642a2cb932176a75d040efea"
Sep 18 16:45:08.118668 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:08.118653    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:45:08.119093 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.119079    2546 scope.go:117] "RemoveContainer" containerID="c84ba3c21c11d4dc01d7be3d0106b61695f39004e266889e7d87594d1f65f2e0"
Sep 18 16:45:08.131731 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.131709    2546 scope.go:117] "RemoveContainer" containerID="7ffa2bf0a0351755ec4c7d243d09d324b5c16d217e2986a9043f161fd394f65d"
Sep 18 16:45:08.143688 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.143670    2546 scope.go:117] "RemoveContainer" containerID="ec7a0640f0844f3a40d039c2f480dfcb228250877b0dee0820cd1c757acd9233"
Sep 18 16:45:08.202530 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:08.202499    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:45:09.129694 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:09.129667    2546 scope.go:117] "RemoveContainer" containerID="f85beff468102e90e392b320d6751adbc123af6c350360ab8f43c9bc0cd0192b"
Sep 18 16:45:09.130098 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:09.129874    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=authorino-operator-76b478bd78-t7kmf_kuadrant-system(f7e45976-cc30-418c-96f4-70e0c667572e)\"" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" podUID="f7e45976-cc30-418c-96f4-70e0c667572e"
Sep 18 16:45:09.903351 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:09.903311    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:45:09.903783 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:09.903765    2546 scope.go:117] "RemoveContainer" containerID="91610200fb7e250c873f96d6871f65b3d42e77ca26047d40e3a7cd89ba1f36cd"
Sep 18 16:45:09.904001 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:09.903982    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-8bfgv_opendatahub(d9b87cf9-1252-4eeb-bd58-194c81ea4a94)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" podUID="d9b87cf9-1252-4eeb-bd58-194c81ea4a94"
Sep 18 16:45:11.882974 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:11.882935    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:45:11.883488 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:11.883422    2546 scope.go:117] "RemoveContainer" containerID="45ddde9ad92050db8a056a7194d049e3904b4e4ecee255bf5af368158417eea1"
Sep 18 16:45:11.883668 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:11.883644    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=odh-model-controller-7998c67dc4-klwkg_opendatahub(4ce26bbe-bd40-4653-a16f-2b0c1c87604b)\"" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" podUID="4ce26bbe-bd40-4653-a16f-2b0c1c87604b"
Sep 18 16:45:12.353999 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:12.353955    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:45:02Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:45:02Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:45:02Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:45:02Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-135-253.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:45:13.995686 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:13.995636    2546 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:45:14.143766 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:14.143725    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:45:14.144181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:14.144165    2546 scope.go:117] "RemoveContainer" containerID="f85beff468102e90e392b320d6751adbc123af6c350360ab8f43c9bc0cd0192b"
Sep 18 16:45:14.144361 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:14.144344    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=authorino-operator-76b478bd78-t7kmf_kuadrant-system(f7e45976-cc30-418c-96f4-70e0c667572e)\"" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" podUID="f7e45976-cc30-418c-96f4-70e0c667572e"
Sep 18 16:45:14.409723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:14.409639    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:45:14.410043 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:14.410010    2546 scope.go:117] "RemoveContainer" containerID="cffca3674c4a9164d4ec101c40d11c66e27300f1642a2cb932176a75d040efea"
Sep 18 16:45:14.410261 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:14.410243    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:45:14.727572 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:14.727540    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:45:14.727921 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:14.727907    2546 scope.go:117] "RemoveContainer" containerID="45ddde9ad92050db8a056a7194d049e3904b4e4ecee255bf5af368158417eea1"
Sep 18 16:45:14.728124 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:14.728104    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=odh-model-controller-7998c67dc4-klwkg_opendatahub(4ce26bbe-bd40-4653-a16f-2b0c1c87604b)\"" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" podUID="4ce26bbe-bd40-4653-a16f-2b0c1c87604b"
Sep 18 16:45:14.754019 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:14.753990    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:45:14.754365 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:14.754351    2546 scope.go:117] "RemoveContainer" containerID="91610200fb7e250c873f96d6871f65b3d42e77ca26047d40e3a7cd89ba1f36cd"
Sep 18 16:45:14.754545 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:14.754530    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=llmisvc-controller-manager-777c6bd7d5-8bfgv_opendatahub(d9b87cf9-1252-4eeb-bd58-194c81ea4a94)\"" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" podUID="d9b87cf9-1252-4eeb-bd58-194c81ea4a94"
Sep 18 16:45:15.182151 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:15.182068    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:45:15.182490 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:15.182408    2546 scope.go:117] "RemoveContainer" containerID="e45640e47570f5a8708515d9979875695e9da23496b279081c84459551368b55"
Sep 18 16:45:15.182590 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:15.182570    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=limitador-operator-controller-manager-74495c69f8-dbsdv_kuadrant-system(61d67412-30af-4fb0-adc0-5251f88b7a41)\"" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" podUID="61d67412-30af-4fb0-adc0-5251f88b7a41"
Sep 18 16:45:16.071580 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:16.071541    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:45:16.071990 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:16.071970    2546 scope.go:117] "RemoveContainer" containerID="e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843"
Sep 18 16:45:16.072218 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:16.072200    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-l7p9n_kuadrant-system(1c53eeff-c125-49c6-ab26-182dd935f398)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podUID="1c53eeff-c125-49c6-ab26-182dd935f398"
Sep 18 16:45:19.131264 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:19.131223    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:45:19.596889 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:19.596857    2546 scope.go:117] "RemoveContainer" containerID="debfef9f6428b8d3ffa9087097c6e36270ed679f4872714ba8b78faf3641950a"
Sep 18 16:45:19.597083 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:19.597014    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:45:22.354535 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:22.354502    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:45:23.450602 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:23.450559    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:45:23.450965 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:23.450939    2546 scope.go:117] "RemoveContainer" containerID="e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843"
Sep 18 16:45:23.451190 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:23.451168    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-l7p9n_kuadrant-system(1c53eeff-c125-49c6-ab26-182dd935f398)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podUID="1c53eeff-c125-49c6-ab26-182dd935f398"
Sep 18 16:45:23.519750 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:23.519719    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:45:23.520076 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:23.520063    2546 scope.go:117] "RemoveContainer" containerID="e45640e47570f5a8708515d9979875695e9da23496b279081c84459551368b55"
Sep 18 16:45:23.520250 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:23.520233    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=limitador-operator-controller-manager-74495c69f8-dbsdv_kuadrant-system(61d67412-30af-4fb0-adc0-5251f88b7a41)\"" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" podUID="61d67412-30af-4fb0-adc0-5251f88b7a41"
Sep 18 16:45:23.721015 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:23.720878    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{kuadrant-operator-controller-manager-5769f99d9f-l7p9n.18d677ac43ee4e3f  kuadrant-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kuadrant-system,Name:kuadrant-operator-controller-manager-5769f99d9f-l7p9n,UID:1c53eeff-c125-49c6-ab26-182dd935f398,APIVersion:v1,ResourceVersion:18673,FieldPath:spec.containers{manager},},Reason:BackOff,Message:Back-off restarting failed container manager in pod kuadrant-operator-controller-manager-5769f99d9f-l7p9n_kuadrant-system(1c53eeff-c125-49c6-ab26-182dd935f398),Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:38:03.684122175 +0000 UTC m=+1979.592902524,LastTimestamp:2026-09-18 16:38:03.684122175 +0000 UTC m=+1979.592902524,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:45:23.881800 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:23.881760    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/healthz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:45:23.996457 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:23.996370    2546 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:45:25.597005 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:25.596976    2546 scope.go:117] "RemoveContainer" containerID="45ddde9ad92050db8a056a7194d049e3904b4e4ecee255bf5af368158417eea1"
Sep 18 16:45:25.597380 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:25.597200    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=odh-model-controller-7998c67dc4-klwkg_opendatahub(4ce26bbe-bd40-4653-a16f-2b0c1c87604b)\"" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" podUID="4ce26bbe-bd40-4653-a16f-2b0c1c87604b"
Sep 18 16:45:26.596958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:26.596926    2546 scope.go:117] "RemoveContainer" containerID="f85beff468102e90e392b320d6751adbc123af6c350360ab8f43c9bc0cd0192b"
Sep 18 16:45:26.597236 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:26.597214    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 20s restarting failed container=manager pod=authorino-operator-76b478bd78-t7kmf_kuadrant-system(f7e45976-cc30-418c-96f4-70e0c667572e)\"" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" podUID="f7e45976-cc30-418c-96f4-70e0c667572e"
Sep 18 16:45:29.130837 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:29.130793    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:45:29.596807 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:29.596780    2546 scope.go:117] "RemoveContainer" containerID="cffca3674c4a9164d4ec101c40d11c66e27300f1642a2cb932176a75d040efea"
Sep 18 16:45:29.596968 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:29.596920    2546 scope.go:117] "RemoveContainer" containerID="91610200fb7e250c873f96d6871f65b3d42e77ca26047d40e3a7cd89ba1f36cd"
Sep 18 16:45:30.238705 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:30.238663    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv" event={"ID":"d9b87cf9-1252-4eeb-bd58-194c81ea4a94","Type":"ContainerStarted","Data":"f09732ce99d302a70e3daf53e40c8556a23aa9a166125359cef445d2fb09b67f"}
Sep 18 16:45:30.239067 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:30.238915    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:45:30.240527 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:30.240504    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerStarted","Data":"a671067901b0ea453c9056b9a1fc669996434959c74951baad5f989629e32231"}
Sep 18 16:45:30.240708 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:30.240693    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:45:32.355341 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:32.355306    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:45:32.596939 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:32.596903    2546 scope.go:117] "RemoveContainer" containerID="debfef9f6428b8d3ffa9087097c6e36270ed679f4872714ba8b78faf3641950a"
Sep 18 16:45:32.597182 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:32.597159    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:45:33.997338 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:33.997295    2546 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:45:35.596721 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:35.596689    2546 scope.go:117] "RemoveContainer" containerID="e45640e47570f5a8708515d9979875695e9da23496b279081c84459551368b55"
Sep 18 16:45:36.273335 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:36.273301    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv" event={"ID":"61d67412-30af-4fb0-adc0-5251f88b7a41","Type":"ContainerStarted","Data":"c031278f3603bac03f545f7f5dbd1212d5f176e966e756cfdccb4f64c93dd726"}
Sep 18 16:45:36.273532 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:36.273508    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:45:36.597618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:36.597528    2546 scope.go:117] "RemoveContainer" containerID="e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843"
Sep 18 16:45:36.597990 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:36.597803    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-l7p9n_kuadrant-system(1c53eeff-c125-49c6-ab26-182dd935f398)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podUID="1c53eeff-c125-49c6-ab26-182dd935f398"
Sep 18 16:45:38.597329 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:38.597288    2546 scope.go:117] "RemoveContainer" containerID="45ddde9ad92050db8a056a7194d049e3904b4e4ecee255bf5af368158417eea1"
Sep 18 16:45:38.597712 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:38.597368    2546 scope.go:117] "RemoveContainer" containerID="f85beff468102e90e392b320d6751adbc123af6c350360ab8f43c9bc0cd0192b"
Sep 18 16:45:39.130973 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:39.130924    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:45:39.293762 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:39.293659    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf" event={"ID":"f7e45976-cc30-418c-96f4-70e0c667572e","Type":"ContainerStarted","Data":"9c90e4255b3464cd3651c83e57cb085d5f5fa7bebd6fae212eb04cce6d30a6e8"}
Sep 18 16:45:39.293960 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:39.293881    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:45:39.295600 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:39.295577    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg" event={"ID":"4ce26bbe-bd40-4653-a16f-2b0c1c87604b","Type":"ContainerStarted","Data":"39a14af95f276df3aaeb4a38c3eb3110dd85dfb9e584bbb49aacb0120cd7ea23"}
Sep 18 16:45:39.295786 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:39.295770    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:45:42.356388 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:42.356345    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:45:43.882386 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:43.882336    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/healthz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:45:43.997567 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:43.997521    2546 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:45:47.281381 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:47.281350    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:45:47.597178 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:47.597097    2546 scope.go:117] "RemoveContainer" containerID="debfef9f6428b8d3ffa9087097c6e36270ed679f4872714ba8b78faf3641950a"
Sep 18 16:45:47.597340 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:47.597293    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:45:49.130567 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:49.130520    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:45:49.597258 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:49.597224    2546 scope.go:117] "RemoveContainer" containerID="e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843"
Sep 18 16:45:50.302739 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:50.302707    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-t7kmf"
Sep 18 16:45:50.303139 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:50.302786    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-klwkg"
Sep 18 16:45:50.355303 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:50.355266    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" event={"ID":"1c53eeff-c125-49c6-ab26-182dd935f398","Type":"ContainerStarted","Data":"4cab227eb4aa20776a26e7c8bc2a9134c691ea7f801b4f02d184b4a53e906965"}
Sep 18 16:45:50.355504 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:50.355488    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:45:52.356819 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:52.356777    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:45:52.356819 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:52.356812    2546 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Sep 18 16:45:53.998420 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:53.998364    2546 controller.go:195] "Failed to update lease" err="Put \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:45:53.998420 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:53.998416    2546 controller.go:115] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease"
Sep 18 16:45:57.725359 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:45:57.725258    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{kserve-controller-manager-d6f4b65b8-pxtkv.18d6764176bff0af  opendatahub   14842 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:kserve-controller-manager-d6f4b65b8-pxtkv,UID:f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7,APIVersion:v1,ResourceVersion:14747,FieldPath:spec.containers{manager},},Reason:Pulling,Message:Pulling image \"quay.io/opendatahub/kserve-controller:odh-v3.5-EA2\",Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:12:05 +0000 UTC,LastTimestamp:2026-09-18 16:38:03.688203 +0000 UTC m=+1979.596983330,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:45:59.130806 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:45:59.130762    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:46:01.248548 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:01.248512    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:46:01.362689 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:01.362658    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:46:01.597207 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:01.597109    2546 scope.go:117] "RemoveContainer" containerID="debfef9f6428b8d3ffa9087097c6e36270ed679f4872714ba8b78faf3641950a"
Sep 18 16:46:02.421364 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:02.421332    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerStarted","Data":"bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"}
Sep 18 16:46:03.882228 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:03.882179    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/healthz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:46:03.882628 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:03.882262    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:46:03.882898 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:03.882872    2546 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"d2e021c90cb80608a2865e381d816aa43dbfbadf33234b051cd216c9f9c5114e"} pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" containerMessage="Container manager failed liveness probe, will be restarted"
Sep 18 16:46:03.882938 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:03.882914    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:46:03.882974 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:03.882924    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" containerID="cri-o://d2e021c90cb80608a2865e381d816aa43dbfbadf33234b051cd216c9f9c5114e" gracePeriod=10
Sep 18 16:46:03.998960 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:03.998918    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="200ms"
Sep 18 16:46:04.433980 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:04.433951    2546 generic.go:356] "Generic (PLEG): container finished" podID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerID="d2e021c90cb80608a2865e381d816aa43dbfbadf33234b051cd216c9f9c5114e" exitCode=1
Sep 18 16:46:04.434157 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:04.434015    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerDied","Data":"d2e021c90cb80608a2865e381d816aa43dbfbadf33234b051cd216c9f9c5114e"}
Sep 18 16:46:04.434157 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:04.434098    2546 scope.go:117] "RemoveContainer" containerID="c28e358e7730fc0f82ecd3c76dbfd56ee4f554326adfdc0446501c801d293b3a"
Sep 18 16:46:04.906615 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:04.906567    2546 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods llmisvc-controller-manager-777c6bd7d5-8bfgv)" podUID="d9b87cf9-1252-4eeb-bd58-194c81ea4a94" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8bfgv"
Sep 18 16:46:05.441866 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:05.441829    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerStarted","Data":"7e7aed4c6cada11ba69d35a4bb8b46a391ca90ba414f1647ac22f3f5b4588c32"}
Sep 18 16:46:05.442121 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:05.442060    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:46:06.287242 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:06.287199    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:46:09.451239 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:09.451189    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:46:11.328218 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:11.328173    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:46:12.603144 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:12.603096    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:46:02Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:46:02Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:46:02Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:46:02Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-135-253.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal/status?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:46:14.199658 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:14.199598    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded" interval="400ms"
Sep 18 16:46:16.369282 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:16.369229    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:46:16.449179 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:16.449137    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:46:19.451199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:19.451147    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:46:21.410179 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:21.410134    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:46:22.604035 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:22.603999    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:46:23.882070 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:23.882009    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/healthz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:46:24.600534 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:24.600486    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded" interval="800ms"
Sep 18 16:46:26.448090 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:26.448045    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:46:26.452163 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:26.452126    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:46:29.450199 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:29.450152    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:46:30.202075 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:30.202016    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": read tcp 10.133.0.2:41860->10.133.0.26:8081: read: connection reset by peer"
Sep 18 16:46:30.580477 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:30.580394    2546 generic.go:356] "Generic (PLEG): container finished" podID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerID="a671067901b0ea453c9056b9a1fc669996434959c74951baad5f989629e32231" exitCode=1
Sep 18 16:46:30.580477 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:30.580465    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerDied","Data":"a671067901b0ea453c9056b9a1fc669996434959c74951baad5f989629e32231"}
Sep 18 16:46:30.580881 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:30.580520    2546 scope.go:117] "RemoveContainer" containerID="cffca3674c4a9164d4ec101c40d11c66e27300f1642a2cb932176a75d040efea"
Sep 18 16:46:30.580925 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:30.580880    2546 scope.go:117] "RemoveContainer" containerID="a671067901b0ea453c9056b9a1fc669996434959c74951baad5f989629e32231"
Sep 18 16:46:30.581170 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:30.581148    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:46:31.728707 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:31.728618    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{odh-model-controller-7998c67dc4-klwkg.18d677ac5e18b209  opendatahub    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:odh-model-controller-7998c67dc4-klwkg,UID:4ce26bbe-bd40-4653-a16f-2b0c1c87604b,APIVersion:v1,ResourceVersion:14179,FieldPath:spec.containers{manager},},Reason:Pulled,Message:Successfully pulled image \"quay.io/opendatahub/odh-model-controller:odh-v3.5-EA2\" in 441ms (441ms including waiting). Image size: 238576181 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:38:04.123107849 +0000 UTC m=+1980.031888192,LastTimestamp:2026-09-18 16:38:04.123107849 +0000 UTC m=+1980.031888192,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:46:32.604741 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:32.604706    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:46:34.410167 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:34.410135    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:46:34.410591 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:34.410491    2546 scope.go:117] "RemoveContainer" containerID="a671067901b0ea453c9056b9a1fc669996434959c74951baad5f989629e32231"
Sep 18 16:46:34.410691 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:34.410672    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:46:35.401162 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:35.401124    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded" interval="1.6s"
Sep 18 16:46:36.448036 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:36.447981    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:46:42.605757 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:42.605723    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:46:43.882443 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:43.882396    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/healthz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:46:46.447726 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:46.447679    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:46:46.597461 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:46.597427    2546 scope.go:117] "RemoveContainer" containerID="a671067901b0ea453c9056b9a1fc669996434959c74951baad5f989629e32231"
Sep 18 16:46:46.597630 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:46.597612    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:46:47.002319 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:47.002283    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded" interval="3.2s"
Sep 18 16:46:52.606016 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:52.605985    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded"
Sep 18 16:46:52.606016 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:46:52.606012    2546 kubelet_node_status.go:523] "Unable to update node status" err="update node status exceeds retry count"
Sep 18 16:46:56.448240 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:46:56.448197    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:47:00.203506 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:00.203446    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded" interval="6.4s"
Sep 18 16:47:00.597063 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:00.596946    2546 scope.go:117] "RemoveContainer" containerID="a671067901b0ea453c9056b9a1fc669996434959c74951baad5f989629e32231"
Sep 18 16:47:00.597224 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:00.597171    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:47:01.746617 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:01.746585    2546 generic.go:356] "Generic (PLEG): container finished" podID="5d260854-98c0-4dd7-89a4-d9b9609d3203" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf" exitCode=1
Sep 18 16:47:01.747008 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:01.746643    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerDied","Data":"bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"}
Sep 18 16:47:01.747008 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:01.746680    2546 scope.go:117] "RemoveContainer" containerID="debfef9f6428b8d3ffa9087097c6e36270ed679f4872714ba8b78faf3641950a"
Sep 18 16:47:01.747116 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:01.747078    2546 scope.go:117] "RemoveContainer" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"
Sep 18 16:47:01.747305 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:01.747285    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:47:03.881873 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:03.881832    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/healthz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:47:03.882285 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:03.881906    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:47:03.882470 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:03.882445    2546 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"7e7aed4c6cada11ba69d35a4bb8b46a391ca90ba414f1647ac22f3f5b4588c32"} pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" containerMessage="Container manager failed liveness probe, will be restarted"
Sep 18 16:47:03.882509 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:03.882494    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" containerID="cri-o://7e7aed4c6cada11ba69d35a4bb8b46a391ca90ba414f1647ac22f3f5b4588c32" gracePeriod=10
Sep 18 16:47:03.882546 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:03.882502    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:47:04.768636 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:04.768601    2546 generic.go:356] "Generic (PLEG): container finished" podID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerID="7e7aed4c6cada11ba69d35a4bb8b46a391ca90ba414f1647ac22f3f5b4588c32" exitCode=1
Sep 18 16:47:04.768819 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:04.768670    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerDied","Data":"7e7aed4c6cada11ba69d35a4bb8b46a391ca90ba414f1647ac22f3f5b4588c32"}
Sep 18 16:47:04.768819 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:04.768726    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerStarted","Data":"7e4c9532ccbc4268324b31b4a91414e6d958287ec6f61f59da86434435756091"}
Sep 18 16:47:04.768819 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:04.768749    2546 scope.go:117] "RemoveContainer" containerID="d2e021c90cb80608a2865e381d816aa43dbfbadf33234b051cd216c9f9c5114e"
Sep 18 16:47:04.769088 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:04.769067    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:47:04.911361 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:04.911319    2546 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws)" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:47:05.733046 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:05.732917    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{kserve-controller-manager-d6f4b65b8-pxtkv.18d677ac61d5360e  opendatahub    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:kserve-controller-manager-d6f4b65b8-pxtkv,UID:f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7,APIVersion:v1,ResourceVersion:14747,FieldPath:spec.containers{manager},},Reason:Pulled,Message:Successfully pulled image \"quay.io/opendatahub/kserve-controller:odh-v3.5-EA2\" in 497ms (497ms including waiting). Image size: 241618392 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:38:04.185794062 +0000 UTC m=+1980.094574406,LastTimestamp:2026-09-18 16:38:04.185794062 +0000 UTC m=+1980.094574406,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:47:12.596871 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:12.596828    2546 scope.go:117] "RemoveContainer" containerID="a671067901b0ea453c9056b9a1fc669996434959c74951baad5f989629e32231"
Sep 18 16:47:12.597387 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:12.596938    2546 scope.go:117] "RemoveContainer" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"
Sep 18 16:47:12.597387 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:12.597135    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:47:13.823340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:13.823303    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerStarted","Data":"c4d645564a463a3a74e921170c04aa55f7adb2a6f1fa5d9a1ef66cbd59ce1232"}
Sep 18 16:47:13.823790 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:13.823528    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:47:15.777945 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:15.777903    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:47:16.603706 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:16.603668    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s"
Sep 18 16:47:23.203000 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:23.202946    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:47:13Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:47:13Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:47:13Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-09-18T16:47:13Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"}]}}\" for node \"ip-10-0-135-253.ec2.internal\": Patch \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal/status?timeout=10s\": context deadline exceeded"
Sep 18 16:47:23.882368 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:23.882326    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/healthz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:47:25.777835 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:25.777791    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:47:26.596869 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:26.596837    2546 scope.go:117] "RemoveContainer" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"
Sep 18 16:47:26.597089 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:26.596999    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:47:33.204180 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:33.204141    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:47:33.605942 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:33.605844    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": context deadline exceeded" interval="7s"
Sep 18 16:47:35.777589 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:35.777546    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:47:39.737781 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:39.737676    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{odh-model-controller-7998c67dc4-klwkg.18d676404bad9249  opendatahub   14461 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:odh-model-controller-7998c67dc4-klwkg,UID:4ce26bbe-bd40-4653-a16f-2b0c1c87604b,APIVersion:v1,ResourceVersion:14179,FieldPath:spec.containers{manager},},Reason:Created,Message:Created container: manager,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:12:00 +0000 UTC,LastTimestamp:2026-09-18 16:38:04.19155842 +0000 UTC m=+1980.100338764,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:47:41.596654 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:41.596624    2546 scope.go:117] "RemoveContainer" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"
Sep 18 16:47:41.597055 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:41.596792    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:47:43.205242 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:43.205204    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:47:43.882402 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:43.882355    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/healthz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:47:45.777554 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:45.777509    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:47:49.451228 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:49.451177    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:47:49.870243 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:49.870151    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:47:50.606691 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:50.606651    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:47:53.205536 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:53.205497    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-253.ec2.internal\": Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/api/v1/nodes/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:47:54.911225 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:54.911175    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:47:55.777445 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:55.777404    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:47:56.597194 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:56.597162    2546 scope.go:117] "RemoveContainer" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"
Sep 18 16:47:56.597576 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:47:56.597349    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:47:59.452173 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:59.452130    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:47:59.452565 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:59.452209    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:47:59.452668 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:59.452650    2546 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"c4d645564a463a3a74e921170c04aa55f7adb2a6f1fa5d9a1ef66cbd59ce1232"} pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" containerMessage="Container manager failed liveness probe, will be restarted"
Sep 18 16:47:59.452706 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:59.452687    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" containerID="cri-o://c4d645564a463a3a74e921170c04aa55f7adb2a6f1fa5d9a1ef66cbd59ce1232" gracePeriod=10
Sep 18 16:47:59.457005 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:59.456970    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": read tcp 10.133.0.2:41842->10.133.0.26:8081: read: connection reset by peer"
Sep 18 16:47:59.457717 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:47:59.457312    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": dial tcp 10.133.0.26:8081: connect: connection refused"
Sep 18 16:48:00.067965 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:00.067931    2546 generic.go:356] "Generic (PLEG): container finished" podID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerID="c4d645564a463a3a74e921170c04aa55f7adb2a6f1fa5d9a1ef66cbd59ce1232" exitCode=2
Sep 18 16:48:00.068182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:00.067987    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerDied","Data":"c4d645564a463a3a74e921170c04aa55f7adb2a6f1fa5d9a1ef66cbd59ce1232"}
Sep 18 16:48:00.068182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:00.068051    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerStarted","Data":"a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"}
Sep 18 16:48:00.068182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:00.068075    2546 scope.go:117] "RemoveContainer" containerID="a671067901b0ea453c9056b9a1fc669996434959c74951baad5f989629e32231"
Sep 18 16:48:00.068347 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:00.068273    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:48:03.881698 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:03.881656    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/healthz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:48:03.882121 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:03.881730    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:48:03.882285 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:03.882262    2546 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"7e4c9532ccbc4268324b31b4a91414e6d958287ec6f61f59da86434435756091"} pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" containerMessage="Container manager failed liveness probe, will be restarted"
Sep 18 16:48:03.882324 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:03.882281    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:48:03.882324 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:03.882311    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" containerID="cri-o://7e4c9532ccbc4268324b31b4a91414e6d958287ec6f61f59da86434435756091" gracePeriod=10
Sep 18 16:48:04.092533 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:04.092501    2546 generic.go:356] "Generic (PLEG): container finished" podID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerID="7e4c9532ccbc4268324b31b4a91414e6d958287ec6f61f59da86434435756091" exitCode=1
Sep 18 16:48:04.092645 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:04.092577    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerDied","Data":"7e4c9532ccbc4268324b31b4a91414e6d958287ec6f61f59da86434435756091"}
Sep 18 16:48:04.092645 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:04.092640    2546 scope.go:117] "RemoveContainer" containerID="7e7aed4c6cada11ba69d35a4bb8b46a391ca90ba414f1647ac22f3f5b4588c32"
Sep 18 16:48:04.913605 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:48:04.913556    2546 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods limitador-operator-controller-manager-74495c69f8-dbsdv)" podUID="61d67412-30af-4fb0-adc0-5251f88b7a41" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dbsdv"
Sep 18 16:48:05.100880 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:05.100829    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerStarted","Data":"e594810d23580cc86595513a4eaf8ebda2a48eea9871f04d01f21685c5652983"}
Sep 18 16:48:05.101070 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:05.100908    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:48:07.607091 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:48:07.607056    2546 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a2ac533c556b142cd9591e189ecb12d9-8c212c476a1563ef.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-135-253.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" interval="7s"
Sep 18 16:48:09.597337 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:09.597301    2546 scope.go:117] "RemoveContainer" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"
Sep 18 16:48:09.597756 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:48:09.597526    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:48:13.741633 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:48:13.741549    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{odh-model-controller-7998c67dc4-klwkg.18d676404c4289e7  opendatahub   14464 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:odh-model-controller-7998c67dc4-klwkg,UID:4ce26bbe-bd40-4653-a16f-2b0c1c87604b,APIVersion:v1,ResourceVersion:14179,FieldPath:spec.containers{manager},},Reason:Started,Message:Started container manager,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:12:00 +0000 UTC,LastTimestamp:2026-09-18 16:38:04.199864512 +0000 UTC m=+1980.108644859,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:48:16.106962 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:16.106923    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:48:23.597481 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:23.597452    2546 scope.go:117] "RemoveContainer" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"
Sep 18 16:48:23.597946 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:48:23.597601    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:48:23.882502 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:23.882421    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/healthz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:48:26.106174 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:26.106137    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:48:36.106326 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:36.106288    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:48:36.117155 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:36.117121    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:48:38.597305 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:38.597204    2546 scope.go:117] "RemoveContainer" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"
Sep 18 16:48:38.602757 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:48:38.597373    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:48:39.450156 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:39.450121    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:48:41.158179 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:41.158138    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:48:43.882128 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:43.882087    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/healthz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:48:46.106616 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:46.106574    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:48:46.199189 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:46.199151    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:48:47.745043 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:48:47.744929    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{kserve-controller-manager-d6f4b65b8-pxtkv.18d676420433e9bb  opendatahub   14900 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:kserve-controller-manager-d6f4b65b8-pxtkv,UID:f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7,APIVersion:v1,ResourceVersion:14747,FieldPath:spec.containers{manager},},Reason:Created,Message:Created container: manager,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:12:07 +0000 UTC,LastTimestamp:2026-09-18 16:38:04.242701517 +0000 UTC m=+1980.151481860,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:48:49.451222 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:49.451177    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:48:51.240220 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:51.240171    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:48:53.597250 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:53.597212    2546 scope.go:117] "RemoveContainer" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"
Sep 18 16:48:53.597654 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:48:53.597411    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:48:56.107049 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:56.106966    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:48:56.281218 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:56.281164    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:48:59.453191 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:48:59.453134    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:49:00.000961 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:00.000916    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.26:8081/readyz\": read tcp 10.133.0.2:54714->10.133.0.26:8081: read: connection reset by peer"
Sep 18 16:49:00.401539 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:00.401443    2546 generic.go:356] "Generic (PLEG): container finished" podID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6" exitCode=1
Sep 18 16:49:00.401539 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:00.401515    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerDied","Data":"a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"}
Sep 18 16:49:00.401754 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:00.401579    2546 scope.go:117] "RemoveContainer" containerID="c4d645564a463a3a74e921170c04aa55f7adb2a6f1fa5d9a1ef66cbd59ce1232"
Sep 18 16:49:00.401934 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:00.401916    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:49:00.402197 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:00.402173    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:49:03.591349 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:03.591313    2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:49:03.882755 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:03.882641    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/healthz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:49:03.882755 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:03.882712    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:49:03.883279 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:03.883261    2546 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"e594810d23580cc86595513a4eaf8ebda2a48eea9871f04d01f21685c5652983"} pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" containerMessage="Container manager failed liveness probe, will be restarted"
Sep 18 16:49:03.883331 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:03.883284    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.20:8081/readyz\": dial tcp 10.133.0.20:8081: connect: connection refused"
Sep 18 16:49:03.883331 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:03.883299    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerName="manager" containerID="cri-o://e594810d23580cc86595513a4eaf8ebda2a48eea9871f04d01f21685c5652983" gracePeriod=10
Sep 18 16:49:04.001330 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:04.001291    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws_opendatahub(eb9ee00b-5814-4425-a7e3-4062562574cc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc"
Sep 18 16:49:04.409609 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:04.409567    2546 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:49:04.410018 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:04.409984    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:49:04.410267 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:04.410248    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:49:04.429296 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:04.429260    2546 generic.go:356] "Generic (PLEG): container finished" podID="eb9ee00b-5814-4425-a7e3-4062562574cc" containerID="e594810d23580cc86595513a4eaf8ebda2a48eea9871f04d01f21685c5652983" exitCode=1
Sep 18 16:49:04.429470 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:04.429336    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerDied","Data":"e594810d23580cc86595513a4eaf8ebda2a48eea9871f04d01f21685c5652983"}
Sep 18 16:49:04.429470 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:04.429395    2546 scope.go:117] "RemoveContainer" containerID="7e4c9532ccbc4268324b31b4a91414e6d958287ec6f61f59da86434435756091"
Sep 18 16:49:04.429839 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:04.429823    2546 scope.go:117] "RemoveContainer" containerID="e594810d23580cc86595513a4eaf8ebda2a48eea9871f04d01f21685c5652983"
Sep 18 16:49:04.430118 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:04.430095    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws_opendatahub(eb9ee00b-5814-4425-a7e3-4062562574cc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc"
Sep 18 16:49:04.916142 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:04.916093    2546 status_manager.go:1018] "Failed to get status for pod" err="the server was unable to return a response in the time allotted, but may still be processing the request (get pods cert-manager-cainjector-559cdd8c87-jcl8r)" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r"
Sep 18 16:49:07.596818 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:07.596787    2546 scope.go:117] "RemoveContainer" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"
Sep 18 16:49:07.597269 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:07.596990    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:49:15.597617 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:15.597577    2546 scope.go:117] "RemoveContainer" containerID="e594810d23580cc86595513a4eaf8ebda2a48eea9871f04d01f21685c5652983"
Sep 18 16:49:15.598106 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:15.597850    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws_opendatahub(eb9ee00b-5814-4425-a7e3-4062562574cc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc"
Sep 18 16:49:17.596518 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:17.596484    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:49:17.596901 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:17.596676    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:49:21.597103 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:21.597066    2546 scope.go:117] "RemoveContainer" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"
Sep 18 16:49:21.597627 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:21.597351    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:49:21.749271 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:21.749168    2546 event.go:359] "Server rejected event (will not retry!)" err="Timeout: request did not complete within requested timeout - context deadline exceeded" event="&Event{ObjectMeta:{kserve-controller-manager-d6f4b65b8-pxtkv.18d676420498933b  opendatahub   14901 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:opendatahub,Name:kserve-controller-manager-d6f4b65b8-pxtkv,UID:f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7,APIVersion:v1,ResourceVersion:14747,FieldPath:spec.containers{manager},},Reason:Started,Message:Started container manager,Source:EventSource{Component:kubelet,Host:ip-10-0-135-253.ec2.internal,},FirstTimestamp:2026-09-18 16:12:07 +0000 UTC,LastTimestamp:2026-09-18 16:38:04.250556773 +0000 UTC m=+1980.159337121,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-253.ec2.internal,}"
Sep 18 16:49:26.596813 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:26.596780    2546 scope.go:117] "RemoveContainer" containerID="e594810d23580cc86595513a4eaf8ebda2a48eea9871f04d01f21685c5652983"
Sep 18 16:49:26.597575 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:26.597537    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws_opendatahub(eb9ee00b-5814-4425-a7e3-4062562574cc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc"
Sep 18 16:49:28.550473 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:28.550438    2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:49:28.975057 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:28.975011    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-89djg\""
Sep 18 16:49:29.020575 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.020544    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bnd84\""
Sep 18 16:49:29.037306 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.037277    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 18 16:49:29.058208 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.058178    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 16:49:29.071210 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.071182    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 18 16:49:29.081121 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.081093    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mdh8x\""
Sep 18 16:49:29.100233 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.100207    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nvjfq\""
Sep 18 16:49:29.118856 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.118828    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 16:49:29.128315 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.128293    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 16:49:29.154792 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.154767    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-p9mkr\""
Sep 18 16:49:29.162748 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.162724    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-47677\""
Sep 18 16:49:29.164037 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.164011    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 16:49:29.167064 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.167045    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 16:49:29.187759 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.187732    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jbsbs\""
Sep 18 16:49:29.194095 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.194077    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-r64dz\""
Sep 18 16:49:29.266554 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.266466    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-b7zzg\""
Sep 18 16:49:29.275512 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.275489    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:49:29.289814 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.289783    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 16:49:29.294723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.294702    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:49:29.329645 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.329614    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 16:49:29.333884 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.333866    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 18 16:49:29.338353 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.338329    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f1pbi65cltjq6\""
Sep 18 16:49:29.339481 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.339465    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hvq9z\""
Sep 18 16:49:29.361232 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.361205    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 16:49:29.381552 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.381522    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vd96t\""
Sep 18 16:49:29.388898 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.388870    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 16:49:29.394909 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.394883    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 18 16:49:29.396474 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.396459    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 16:49:29.406826 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.406805    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 16:49:29.407611 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.407595    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rdd2r\""
Sep 18 16:49:29.411975 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.411957    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mvfv2\""
Sep 18 16:49:29.422100 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.422082    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 16:49:29.438228 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.438203    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tjzzn\""
Sep 18 16:49:29.446829 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.446810    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jdlq5\""
Sep 18 16:49:29.448970 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.448955    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 18 16:49:29.455782 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.455764    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 16:49:29.474449 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.474425    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-a75117-openshift-default-dockercfg-x6fzh\""
Sep 18 16:49:29.479150 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.479128    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wckmk\""
Sep 18 16:49:29.485710 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.485694    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mnksp\""
Sep 18 16:49:29.486103 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.486082    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-11725e-openshift-default-dockercfg-btfmq\""
Sep 18 16:49:29.495508 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.495490    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8cf7l\""
Sep 18 16:49:29.510329 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.510309    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kcclj\""
Sep 18 16:49:29.518118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.518065    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"rate-test-model-a75117-kserve-self-signed-certs\""
Sep 18 16:49:29.518198 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.518114    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-pfgg2\""
Sep 18 16:49:29.520324 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.520307    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 16:49:29.532405 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.532380    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-a75117\"/\"default-dockercfg-jshkt\""
Sep 18 16:49:29.570743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.570717    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 16:49:29.589992 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.589962    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6qnc4\""
Sep 18 16:49:29.610618 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.610586    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8cs6v\""
Sep 18 16:49:29.629321 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.629284    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-phlgp\""
Sep 18 16:49:29.630384 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.630369    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vx59n\""
Sep 18 16:49:29.633939 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.633922    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j9g75\""
Sep 18 16:49:29.654184 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.654163    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-h4tf4\""
Sep 18 16:49:29.669619 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.669592    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 18 16:49:29.685365 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.685335    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kgw4g\""
Sep 18 16:49:29.686644 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.686626    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tzx5z\""
Sep 18 16:49:29.687131 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.687116    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bm6xr\""
Sep 18 16:49:29.690055 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.690012    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 16:49:29.696252 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.696232    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-snrgd\""
Sep 18 16:49:29.697259 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.697240    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 16:49:29.706411 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.706392    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zcx4s\""
Sep 18 16:49:29.715555 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.715532    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dc6ds\""
Sep 18 16:49:29.716694 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.716677    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 18 16:49:29.720198 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.720180    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bxqxp\""
Sep 18 16:49:29.726723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.726706    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 16:49:29.736290 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.736272    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sr7qq\""
Sep 18 16:49:29.750513 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.750492    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vf5nr\""
Sep 18 16:49:29.761257 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.761233    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 16:49:29.763572 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.763555    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-69r9k\""
Sep 18 16:49:29.766454 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.766440    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-p7v7w\""
Sep 18 16:49:29.766773 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.766760    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 18 16:49:29.779187 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.779131    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bgqnj\""
Sep 18 16:49:29.799597 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.799577    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 18 16:49:29.801172 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.801158    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 16:49:29.815232 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.815212    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h4v9d\""
Sep 18 16:49:29.815331 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.815320    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 16:49:29.823814 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:29.823794    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 16:49:32.597219 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:32.597189    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:49:32.597616 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:32.597425    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:49:36.596918 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:36.596881    2546 scope.go:117] "RemoveContainer" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"
Sep 18 16:49:36.597478 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:36.597117    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-cainjector\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=cert-manager-cainjector pod=cert-manager-cainjector-559cdd8c87-jcl8r_cert-manager(5d260854-98c0-4dd7-89a4-d9b9609d3203)\"" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" podUID="5d260854-98c0-4dd7-89a4-d9b9609d3203"
Sep 18 16:49:38.597177 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:38.597136    2546 scope.go:117] "RemoveContainer" containerID="e594810d23580cc86595513a4eaf8ebda2a48eea9871f04d01f21685c5652983"
Sep 18 16:49:38.597598 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:38.597370    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws_opendatahub(eb9ee00b-5814-4425-a7e3-4062562574cc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc"
Sep 18 16:49:44.599632 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:44.599596    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:49:44.600198 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:44.599880    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:49:51.597149 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:51.597118    2546 scope.go:117] "RemoveContainer" containerID="bbf89fb2c35a70e4f037414544a6a4c8f170ec1b53b3877c1f5ea429945e5eaf"
Sep 18 16:49:51.687164 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:51.687133    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-jcl8r" event={"ID":"5d260854-98c0-4dd7-89a4-d9b9609d3203","Type":"ContainerStarted","Data":"7c4f331ddb57ac935626b978047161bb644b38f63c8d0fb93ad2457b720a9345"}
Sep 18 16:49:52.596932 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:52.596888    2546 scope.go:117] "RemoveContainer" containerID="e594810d23580cc86595513a4eaf8ebda2a48eea9871f04d01f21685c5652983"
Sep 18 16:49:52.597203 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:52.597176    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws_opendatahub(eb9ee00b-5814-4425-a7e3-4062562574cc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc"
Sep 18 16:49:55.597013 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:49:55.596971    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:49:55.597455 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:49:55.597191    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:50:05.098615 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:05.098546    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-55f6fc7b86-7xldd_45b59ace-62a2-45ae-9f58-a3d21c8c2ab5/service-ca-controller/1.log"
Sep 18 16:50:05.099194 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:05.099178    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57cdc4b768-7sgr7_8e62eede-ea1b-415e-8549-3df93215459e/service-ca-operator/2.log"
Sep 18 16:50:05.109047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:05.109006    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-55f6fc7b86-7xldd_45b59ace-62a2-45ae-9f58-a3d21c8c2ab5/service-ca-controller/1.log"
Sep 18 16:50:05.109641 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:05.109628    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57cdc4b768-7sgr7_8e62eede-ea1b-415e-8549-3df93215459e/service-ca-operator/2.log"
Sep 18 16:50:05.597140 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:05.597114    2546 scope.go:117] "RemoveContainer" containerID="e594810d23580cc86595513a4eaf8ebda2a48eea9871f04d01f21685c5652983"
Sep 18 16:50:05.597353 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:50:05.597333    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws_opendatahub(eb9ee00b-5814-4425-a7e3-4062562574cc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc"
Sep 18 16:50:07.597242 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:07.597211    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:50:07.597617 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:50:07.597409    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:50:10.631919 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:10.631883    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829165-sddfk"]
Sep 18 16:50:10.636137 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:10.636108    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829165-kxvjt"]
Sep 18 16:50:10.636314 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:10.636252    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829165-sddfk"
Sep 18 16:50:10.639683 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:10.639658    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829165-kxvjt"
Sep 18 16:50:10.640676 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:10.640648    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-g5ssz\""
Sep 18 16:50:10.649113 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:10.649089    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829165-sddfk"
Sep 18 16:50:10.649371 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:10.649254    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829165-sddfk"]
Sep 18 16:50:10.655253 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:10.655221    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829165-kxvjt"]
Sep 18 16:50:10.659738 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:10.659712    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829165-kxvjt"
Sep 18 16:50:10.962003 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:10.961968    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829165-sddfk"]
Sep 18 16:50:10.965370 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:50:10.965325    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7cf6fba_8b83_4edb_9c6d_8e9fada753f1.slice/crio-95f21c002573d384210cd35478399834d953eaa3853dc65aeabeab499ac7c2f6 WatchSource:0}: Error finding container 95f21c002573d384210cd35478399834d953eaa3853dc65aeabeab499ac7c2f6: Status 404 returned error can't find the container with id 95f21c002573d384210cd35478399834d953eaa3853dc65aeabeab499ac7c2f6
Sep 18 16:50:10.975693 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:10.975664    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829165-kxvjt"]
Sep 18 16:50:10.980068 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:50:10.980008    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00fd175_7121_417a_8eb0_badc33e0ae8f.slice/crio-e3d722b5ddfa08026f04b3cf1990656bfafb5dd941503f89e32231f2659a48bd WatchSource:0}: Error finding container e3d722b5ddfa08026f04b3cf1990656bfafb5dd941503f89e32231f2659a48bd: Status 404 returned error can't find the container with id e3d722b5ddfa08026f04b3cf1990656bfafb5dd941503f89e32231f2659a48bd
Sep 18 16:50:11.800569 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:11.800521    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829165-kxvjt" event={"ID":"f00fd175-7121-417a-8eb0-badc33e0ae8f","Type":"ContainerStarted","Data":"27b90d415b8f4f2ca02667d95ee01dd1a988b4d4551a4af4cd851b9a5906ab83"}
Sep 18 16:50:11.801068 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:11.800591    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829165-kxvjt" event={"ID":"f00fd175-7121-417a-8eb0-badc33e0ae8f","Type":"ContainerStarted","Data":"e3d722b5ddfa08026f04b3cf1990656bfafb5dd941503f89e32231f2659a48bd"}
Sep 18 16:50:11.802261 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:11.802227    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829165-sddfk" event={"ID":"a7cf6fba-8b83-4edb-9c6d-8e9fada753f1","Type":"ContainerStarted","Data":"0641c1f680544052fb3acd6a02914df1bdd8cb6f87b035a77685827704c44e15"}
Sep 18 16:50:11.802364 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:11.802277    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829165-sddfk" event={"ID":"a7cf6fba-8b83-4edb-9c6d-8e9fada753f1","Type":"ContainerStarted","Data":"95f21c002573d384210cd35478399834d953eaa3853dc65aeabeab499ac7c2f6"}
Sep 18 16:50:11.854340 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:11.854286    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829165-kxvjt" podStartSLOduration=1.85426618 podStartE2EDuration="1.85426618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:50:11.825366046 +0000 UTC m=+2707.734146398" watchObservedRunningTime="2026-09-18 16:50:11.85426618 +0000 UTC m=+2707.763046531"
Sep 18 16:50:11.855162 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:11.855130    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829165-sddfk" podStartSLOduration=1.8551215380000001 podStartE2EDuration="1.855121538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:50:11.853715593 +0000 UTC m=+2707.762495955" watchObservedRunningTime="2026-09-18 16:50:11.855121538 +0000 UTC m=+2707.763901887"
Sep 18 16:50:12.810753 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:12.810711    2546 generic.go:356] "Generic (PLEG): container finished" podID="a7cf6fba-8b83-4edb-9c6d-8e9fada753f1" containerID="0641c1f680544052fb3acd6a02914df1bdd8cb6f87b035a77685827704c44e15" exitCode=0
Sep 18 16:50:12.811205 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:12.810809    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829165-sddfk" event={"ID":"a7cf6fba-8b83-4edb-9c6d-8e9fada753f1","Type":"ContainerDied","Data":"0641c1f680544052fb3acd6a02914df1bdd8cb6f87b035a77685827704c44e15"}
Sep 18 16:50:12.812491 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:12.812466    2546 generic.go:356] "Generic (PLEG): container finished" podID="f00fd175-7121-417a-8eb0-badc33e0ae8f" containerID="27b90d415b8f4f2ca02667d95ee01dd1a988b4d4551a4af4cd851b9a5906ab83" exitCode=0
Sep 18 16:50:12.812609 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:12.812550    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829165-kxvjt" event={"ID":"f00fd175-7121-417a-8eb0-badc33e0ae8f","Type":"ContainerDied","Data":"27b90d415b8f4f2ca02667d95ee01dd1a988b4d4551a4af4cd851b9a5906ab83"}
Sep 18 16:50:13.962687 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:13.962649    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829165-sddfk"
Sep 18 16:50:13.988323 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:13.988294    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829165-kxvjt"
Sep 18 16:50:14.824877 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:14.824841    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829165-sddfk"
Sep 18 16:50:14.825083 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:14.824887    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829165-sddfk" event={"ID":"a7cf6fba-8b83-4edb-9c6d-8e9fada753f1","Type":"ContainerDied","Data":"95f21c002573d384210cd35478399834d953eaa3853dc65aeabeab499ac7c2f6"}
Sep 18 16:50:14.825083 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:14.824955    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95f21c002573d384210cd35478399834d953eaa3853dc65aeabeab499ac7c2f6"
Sep 18 16:50:14.826585 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:14.826562    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829165-kxvjt"
Sep 18 16:50:14.826712 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:14.826569    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829165-kxvjt" event={"ID":"f00fd175-7121-417a-8eb0-badc33e0ae8f","Type":"ContainerDied","Data":"e3d722b5ddfa08026f04b3cf1990656bfafb5dd941503f89e32231f2659a48bd"}
Sep 18 16:50:14.826712 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:14.826675    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3d722b5ddfa08026f04b3cf1990656bfafb5dd941503f89e32231f2659a48bd"
Sep 18 16:50:18.597538 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:18.597506    2546 scope.go:117] "RemoveContainer" containerID="e594810d23580cc86595513a4eaf8ebda2a48eea9871f04d01f21685c5652983"
Sep 18 16:50:18.597926 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:50:18.597711    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=manager pod=opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws_opendatahub(eb9ee00b-5814-4425-a7e3-4062562574cc)\"" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" podUID="eb9ee00b-5814-4425-a7e3-4062562574cc"
Sep 18 16:50:20.597400 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:20.597361    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:50:20.597809 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:50:20.597633    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:50:33.597239 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:33.597193    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:50:33.597642 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:33.597280    2546 scope.go:117] "RemoveContainer" containerID="e594810d23580cc86595513a4eaf8ebda2a48eea9871f04d01f21685c5652983"
Sep 18 16:50:33.597642 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:50:33.597463    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:50:33.597642 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:33.597609    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:50:34.942437 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:34.942402    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws" event={"ID":"eb9ee00b-5814-4425-a7e3-4062562574cc","Type":"ContainerStarted","Data":"f49c1bc4b1458d64bf6ad52f963c12eb474e18614b061a41ece9c76f44c14d41"}
Sep 18 16:50:34.942864 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:34.942715    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:50:44.598952 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:44.598921    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:50:44.599601 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:50:44.599143    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:50:45.950300 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:45.950267    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws"
Sep 18 16:50:56.596938 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:50:56.596902    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:50:56.597347 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:50:56.597194    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:51:09.597369 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:51:09.597334    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:51:09.597775 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:51:09.597546    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:51:20.596975 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:51:20.596937    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:51:20.597519 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:51:20.597225    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:51:31.597205 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:51:31.597127    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:51:31.597592 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:51:31.597326    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=manager pod=kserve-controller-manager-d6f4b65b8-pxtkv_opendatahub(f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7)\"" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" podUID="f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7"
Sep 18 16:51:43.596899 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:51:43.596865    2546 scope.go:117] "RemoveContainer" containerID="a3549a10bf47dd946eb7e7c79e287dfaf379a3858477410e6286f4e6f86255c6"
Sep 18 16:51:44.338863 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:51:44.338822    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv" event={"ID":"f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7","Type":"ContainerStarted","Data":"007d00d09755fac87a0a25032b34456f992e321b24eb926ff615751f73d8a541"}
Sep 18 16:51:44.339210 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:51:44.339186    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:51:53.963128 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:51:53.963089    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:52:03.669643 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:03.669603    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:52:15.348497 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:15.348462    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-pxtkv"
Sep 18 16:52:18.745886 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:18.745852    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:52:36.694435 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.694384    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-mz7px"]
Sep 18 16:52:36.694894 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.694788    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7cf6fba-8b83-4edb-9c6d-8e9fada753f1" containerName="cleanup"
Sep 18 16:52:36.694894 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.694800    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7cf6fba-8b83-4edb-9c6d-8e9fada753f1" containerName="cleanup"
Sep 18 16:52:36.694894 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.694822    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f00fd175-7121-417a-8eb0-badc33e0ae8f" containerName="cleanup"
Sep 18 16:52:36.694894 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.694828    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f00fd175-7121-417a-8eb0-badc33e0ae8f" containerName="cleanup"
Sep 18 16:52:36.695047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.694900    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f00fd175-7121-417a-8eb0-badc33e0ae8f" containerName="cleanup"
Sep 18 16:52:36.695047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.694911    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7cf6fba-8b83-4edb-9c6d-8e9fada753f1" containerName="cleanup"
Sep 18 16:52:36.698124 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.698099    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:36.701616 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.701588    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-sysctl-allowlist\""
Sep 18 16:52:36.733969 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.733920    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxd8m\" (UniqueName: \"kubernetes.io/projected/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-kube-api-access-qxd8m\") pod \"cni-sysctl-allowlist-ds-mz7px\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") " pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:36.734171 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.733975    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-mz7px\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") " pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:36.734171 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.734095    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-mz7px\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") " pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:36.734171 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.734132    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-ready\") pod \"cni-sysctl-allowlist-ds-mz7px\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") " pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:36.835186 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.835142    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxd8m\" (UniqueName: \"kubernetes.io/projected/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-kube-api-access-qxd8m\") pod \"cni-sysctl-allowlist-ds-mz7px\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") " pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:36.835186 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.835191    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-mz7px\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") " pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:36.835455 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.835260    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-mz7px\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") " pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:36.835455 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.835290    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-ready\") pod \"cni-sysctl-allowlist-ds-mz7px\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") " pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:36.835455 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.835366    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-tuning-conf-dir\") pod \"cni-sysctl-allowlist-ds-mz7px\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") " pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:36.835674 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.835658    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-ready\") pod \"cni-sysctl-allowlist-ds-mz7px\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") " pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:36.835823 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.835803    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-cni-sysctl-allowlist\") pod \"cni-sysctl-allowlist-ds-mz7px\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") " pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:36.851510 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:36.851480    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxd8m\" (UniqueName: \"kubernetes.io/projected/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-kube-api-access-qxd8m\") pod \"cni-sysctl-allowlist-ds-mz7px\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") " pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:37.007628 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:37.007533    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:37.020310 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:52:37.020277    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8dd4c6b_8ac9_4c97_b807_3b8d955802df.slice/crio-414d8706b6d4b91683dddd75fcb6c0f1e57fc082562672daa0fe1e2ba0dd1dc1 WatchSource:0}: Error finding container 414d8706b6d4b91683dddd75fcb6c0f1e57fc082562672daa0fe1e2ba0dd1dc1: Status 404 returned error can't find the container with id 414d8706b6d4b91683dddd75fcb6c0f1e57fc082562672daa0fe1e2ba0dd1dc1
Sep 18 16:52:37.583384 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:37.583339    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:52:37.627945 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:37.627904    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px" event={"ID":"d8dd4c6b-8ac9-4c97-b807-3b8d955802df","Type":"ContainerStarted","Data":"bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17"}
Sep 18 16:52:37.628142 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:37.627964    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px" event={"ID":"d8dd4c6b-8ac9-4c97-b807-3b8d955802df","Type":"ContainerStarted","Data":"414d8706b6d4b91683dddd75fcb6c0f1e57fc082562672daa0fe1e2ba0dd1dc1"}
Sep 18 16:52:37.628194 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:37.628157    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:37.652689 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:37.652639    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px" podStartSLOduration=1.652623546 podStartE2EDuration="1.652623546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:52:37.650876507 +0000 UTC m=+2853.559656879" watchObservedRunningTime="2026-09-18 16:52:37.652623546 +0000 UTC m=+2853.561403895"
Sep 18 16:52:38.646694 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:38.646666    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:52:39.678555 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:39.678511    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-mz7px"]
Sep 18 16:52:40.647058 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:40.647005    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px" podUID="d8dd4c6b-8ac9-4c97-b807-3b8d955802df" containerName="kube-multus-additional-cni-plugins" containerID="cri-o://bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17" gracePeriod=30
Sep 18 16:52:46.118976 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:46.118941    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"]
Sep 18 16:52:46.119432 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:46.119168    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager" containerID="cri-o://4cab227eb4aa20776a26e7c8bc2a9134c691ea7f801b4f02d184b4a53e906965" gracePeriod=10
Sep 18 16:52:48.470671 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.470645    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:52:48.558224 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.558135    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxzwn\" (UniqueName: \"kubernetes.io/projected/1c53eeff-c125-49c6-ab26-182dd935f398-kube-api-access-xxzwn\") pod \"1c53eeff-c125-49c6-ab26-182dd935f398\" (UID: \"1c53eeff-c125-49c6-ab26-182dd935f398\") "
Sep 18 16:52:48.558389 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.558292    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c53eeff-c125-49c6-ab26-182dd935f398-extensions-socket-volume\") pod \"1c53eeff-c125-49c6-ab26-182dd935f398\" (UID: \"1c53eeff-c125-49c6-ab26-182dd935f398\") "
Sep 18 16:52:48.558648 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.558621    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c53eeff-c125-49c6-ab26-182dd935f398-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1c53eeff-c125-49c6-ab26-182dd935f398" (UID: "1c53eeff-c125-49c6-ab26-182dd935f398"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:52:48.560242 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.560222    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c53eeff-c125-49c6-ab26-182dd935f398-kube-api-access-xxzwn" (OuterVolumeSpecName: "kube-api-access-xxzwn") pod "1c53eeff-c125-49c6-ab26-182dd935f398" (UID: "1c53eeff-c125-49c6-ab26-182dd935f398"). InnerVolumeSpecName "kube-api-access-xxzwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:52:48.637948 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:52:48.637909    2546 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:52:48.638942 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:52:48.638911    2546 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:52:48.639792 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:52:48.639761    2546 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:52:48.639877 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:52:48.639803    2546 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px" podUID="d8dd4c6b-8ac9-4c97-b807-3b8d955802df" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Sep 18 16:52:48.660235 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.660204    2546 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c53eeff-c125-49c6-ab26-182dd935f398-extensions-socket-volume\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:52:48.660235 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.660232    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxzwn\" (UniqueName: \"kubernetes.io/projected/1c53eeff-c125-49c6-ab26-182dd935f398-kube-api-access-xxzwn\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:52:48.691647 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.691614    2546 generic.go:356] "Generic (PLEG): container finished" podID="1c53eeff-c125-49c6-ab26-182dd935f398" containerID="4cab227eb4aa20776a26e7c8bc2a9134c691ea7f801b4f02d184b4a53e906965" exitCode=0
Sep 18 16:52:48.691838 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.691664    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"
Sep 18 16:52:48.691838 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.691699    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" event={"ID":"1c53eeff-c125-49c6-ab26-182dd935f398","Type":"ContainerDied","Data":"4cab227eb4aa20776a26e7c8bc2a9134c691ea7f801b4f02d184b4a53e906965"}
Sep 18 16:52:48.691838 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.691764    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n" event={"ID":"1c53eeff-c125-49c6-ab26-182dd935f398","Type":"ContainerDied","Data":"ee13b876ea57adbf3d8e685360b70ffd8861eaebd5822d42ee419feb6edc5614"}
Sep 18 16:52:48.691838 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.691800    2546 scope.go:117] "RemoveContainer" containerID="4cab227eb4aa20776a26e7c8bc2a9134c691ea7f801b4f02d184b4a53e906965"
Sep 18 16:52:48.700723 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.700706    2546 scope.go:117] "RemoveContainer" containerID="e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843"
Sep 18 16:52:48.708731 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.708714    2546 scope.go:117] "RemoveContainer" containerID="4cab227eb4aa20776a26e7c8bc2a9134c691ea7f801b4f02d184b4a53e906965"
Sep 18 16:52:48.709037 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:52:48.709006    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cab227eb4aa20776a26e7c8bc2a9134c691ea7f801b4f02d184b4a53e906965\": container with ID starting with 4cab227eb4aa20776a26e7c8bc2a9134c691ea7f801b4f02d184b4a53e906965 not found: ID does not exist" containerID="4cab227eb4aa20776a26e7c8bc2a9134c691ea7f801b4f02d184b4a53e906965"
Sep 18 16:52:48.709108 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.709048    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cab227eb4aa20776a26e7c8bc2a9134c691ea7f801b4f02d184b4a53e906965"} err="failed to get container status \"4cab227eb4aa20776a26e7c8bc2a9134c691ea7f801b4f02d184b4a53e906965\": rpc error: code = NotFound desc = could not find container \"4cab227eb4aa20776a26e7c8bc2a9134c691ea7f801b4f02d184b4a53e906965\": container with ID starting with 4cab227eb4aa20776a26e7c8bc2a9134c691ea7f801b4f02d184b4a53e906965 not found: ID does not exist"
Sep 18 16:52:48.709108 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.709065    2546 scope.go:117] "RemoveContainer" containerID="e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843"
Sep 18 16:52:48.709321 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:52:48.709302    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843\": container with ID starting with e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843 not found: ID does not exist" containerID="e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843"
Sep 18 16:52:48.709420 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.709331    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843"} err="failed to get container status \"e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843\": rpc error: code = NotFound desc = could not find container \"e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843\": container with ID starting with e22bfe1fbadd40f8053067177780d7a0ef54c16e5d42b1daaa9b6a3157b04843 not found: ID does not exist"
Sep 18 16:52:48.714072 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.714051    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"]
Sep 18 16:52:48.721058 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:48.720233    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l7p9n"]
Sep 18 16:52:50.603144 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:52:50.603113    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" path="/var/lib/kubelet/pods/1c53eeff-c125-49c6-ab26-182dd935f398/volumes"
Sep 18 16:52:58.638165 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:52:58.638077    2546 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:52:58.639207 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:52:58.639160    2546 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:52:58.640049 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:52:58.640008    2546 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:52:58.640114 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:52:58.640062    2546 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px" podUID="d8dd4c6b-8ac9-4c97-b807-3b8d955802df" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Sep 18 16:53:08.637604 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:53:08.637559    2546 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:53:08.638619 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:53:08.638585    2546 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:53:08.639572 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:53:08.639541    2546 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17" cmd=["/bin/bash","-c","test -f /ready/ready"]
Sep 18 16:53:08.639691 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:53:08.639579    2546 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px" podUID="d8dd4c6b-8ac9-4c97-b807-3b8d955802df" containerName="kube-multus-additional-cni-plugins" probeResult="unknown"
Sep 18 16:53:10.727916 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.727888    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-mz7px_d8dd4c6b-8ac9-4c97-b807-3b8d955802df/kube-multus-additional-cni-plugins/0.log"
Sep 18 16:53:10.730078 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.730060    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:53:10.785209 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.785174    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-tuning-conf-dir\") pod \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") "
Sep 18 16:53:10.785409 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.785276    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-tuning-conf-dir" (OuterVolumeSpecName: "tuning-conf-dir") pod "d8dd4c6b-8ac9-4c97-b807-3b8d955802df" (UID: "d8dd4c6b-8ac9-4c97-b807-3b8d955802df"). InnerVolumeSpecName "tuning-conf-dir". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 18 16:53:10.785409 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.785288    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-ready\") pod \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") "
Sep 18 16:53:10.785409 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.785363    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-cni-sysctl-allowlist\") pod \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") "
Sep 18 16:53:10.785409 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.785389    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxd8m\" (UniqueName: \"kubernetes.io/projected/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-kube-api-access-qxd8m\") pod \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\" (UID: \"d8dd4c6b-8ac9-4c97-b807-3b8d955802df\") "
Sep 18 16:53:10.785674 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.785599    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-ready" (OuterVolumeSpecName: "ready") pod "d8dd4c6b-8ac9-4c97-b807-3b8d955802df" (UID: "d8dd4c6b-8ac9-4c97-b807-3b8d955802df"). InnerVolumeSpecName "ready". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:10.785755 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.785729    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-cni-sysctl-allowlist" (OuterVolumeSpecName: "cni-sysctl-allowlist") pod "d8dd4c6b-8ac9-4c97-b807-3b8d955802df" (UID: "d8dd4c6b-8ac9-4c97-b807-3b8d955802df"). InnerVolumeSpecName "cni-sysctl-allowlist". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:53:10.785802 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.785742    2546 reconciler_common.go:299] "Volume detached for volume \"ready\" (UniqueName: \"kubernetes.io/empty-dir/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-ready\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:53:10.785802 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.785783    2546 reconciler_common.go:299] "Volume detached for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-tuning-conf-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:53:10.787779 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.787757    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-kube-api-access-qxd8m" (OuterVolumeSpecName: "kube-api-access-qxd8m") pod "d8dd4c6b-8ac9-4c97-b807-3b8d955802df" (UID: "d8dd4c6b-8ac9-4c97-b807-3b8d955802df"). InnerVolumeSpecName "kube-api-access-qxd8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:53:10.810020 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.809992    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_cni-sysctl-allowlist-ds-mz7px_d8dd4c6b-8ac9-4c97-b807-3b8d955802df/kube-multus-additional-cni-plugins/0.log"
Sep 18 16:53:10.810211 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.810047    2546 generic.go:356] "Generic (PLEG): container finished" podID="d8dd4c6b-8ac9-4c97-b807-3b8d955802df" containerID="bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17" exitCode=137
Sep 18 16:53:10.810211 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.810123    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px"
Sep 18 16:53:10.810211 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.810157    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px" event={"ID":"d8dd4c6b-8ac9-4c97-b807-3b8d955802df","Type":"ContainerDied","Data":"bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17"}
Sep 18 16:53:10.810324 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.810216    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/cni-sysctl-allowlist-ds-mz7px" event={"ID":"d8dd4c6b-8ac9-4c97-b807-3b8d955802df","Type":"ContainerDied","Data":"414d8706b6d4b91683dddd75fcb6c0f1e57fc082562672daa0fe1e2ba0dd1dc1"}
Sep 18 16:53:10.810324 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.810238    2546 scope.go:117] "RemoveContainer" containerID="bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17"
Sep 18 16:53:10.820345 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.820322    2546 scope.go:117] "RemoveContainer" containerID="bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17"
Sep 18 16:53:10.820676 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:53:10.820653    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17\": container with ID starting with bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17 not found: ID does not exist" containerID="bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17"
Sep 18 16:53:10.820731 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.820690    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17"} err="failed to get container status \"bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17\": rpc error: code = NotFound desc = could not find container \"bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17\": container with ID starting with bc044570bf96c51839d5c1a3f8634c15ea2da0095aa021c652527838e6400f17 not found: ID does not exist"
Sep 18 16:53:10.833591 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.833566    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-mz7px"]
Sep 18 16:53:10.836527 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.836500    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-multus/cni-sysctl-allowlist-ds-mz7px"]
Sep 18 16:53:10.887319 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.887279    2546 reconciler_common.go:299] "Volume detached for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-cni-sysctl-allowlist\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:53:10.887319 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:10.887312    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxd8m\" (UniqueName: \"kubernetes.io/projected/d8dd4c6b-8ac9-4c97-b807-3b8d955802df-kube-api-access-qxd8m\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:53:12.603277 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:12.603243    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8dd4c6b-8ac9-4c97-b807-3b8d955802df" path="/var/lib/kubelet/pods/d8dd4c6b-8ac9-4c97-b807-3b8d955802df/volumes"
Sep 18 16:53:46.842136 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842096    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm"]
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842464    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8dd4c6b-8ac9-4c97-b807-3b8d955802df" containerName="kube-multus-additional-cni-plugins"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842475    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8dd4c6b-8ac9-4c97-b807-3b8d955802df" containerName="kube-multus-additional-cni-plugins"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842487    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842493    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842499    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842506    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842515    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842520    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842526    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842531    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842544    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842551    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842610    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842623    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842629    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842634    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:53:46.844160 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.842639    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8dd4c6b-8ac9-4c97-b807-3b8d955802df" containerName="kube-multus-additional-cni-plugins"
Sep 18 16:53:46.845519 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.845502    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm"
Sep 18 16:53:46.850208 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.850184    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jdlq5\""
Sep 18 16:53:46.873054 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.873003    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm"]
Sep 18 16:53:46.943581 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.943545    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c142618b-0b7a-4b05-b41c-dffa1de8cae2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z9xzm\" (UID: \"c142618b-0b7a-4b05-b41c-dffa1de8cae2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm"
Sep 18 16:53:46.943581 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:46.943591    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qg8q\" (UniqueName: \"kubernetes.io/projected/c142618b-0b7a-4b05-b41c-dffa1de8cae2-kube-api-access-5qg8q\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z9xzm\" (UID: \"c142618b-0b7a-4b05-b41c-dffa1de8cae2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm"
Sep 18 16:53:47.044464 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:47.044421    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c142618b-0b7a-4b05-b41c-dffa1de8cae2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z9xzm\" (UID: \"c142618b-0b7a-4b05-b41c-dffa1de8cae2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm"
Sep 18 16:53:47.044464 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:47.044469    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qg8q\" (UniqueName: \"kubernetes.io/projected/c142618b-0b7a-4b05-b41c-dffa1de8cae2-kube-api-access-5qg8q\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z9xzm\" (UID: \"c142618b-0b7a-4b05-b41c-dffa1de8cae2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm"
Sep 18 16:53:47.044824 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:47.044802    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c142618b-0b7a-4b05-b41c-dffa1de8cae2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z9xzm\" (UID: \"c142618b-0b7a-4b05-b41c-dffa1de8cae2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm"
Sep 18 16:53:47.056466 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:47.056430    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qg8q\" (UniqueName: \"kubernetes.io/projected/c142618b-0b7a-4b05-b41c-dffa1de8cae2-kube-api-access-5qg8q\") pod \"kuadrant-operator-controller-manager-5769f99d9f-z9xzm\" (UID: \"c142618b-0b7a-4b05-b41c-dffa1de8cae2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm"
Sep 18 16:53:47.156361 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:47.156260    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm"
Sep 18 16:53:47.442357 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:47.442331    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm"]
Sep 18 16:53:47.444515 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:53:47.444486    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc142618b_0b7a_4b05_b41c_dffa1de8cae2.slice/crio-080805eb4b018aef29a4a5302846eb0346728c1fd900a9206afaf874ecd4af14 WatchSource:0}: Error finding container 080805eb4b018aef29a4a5302846eb0346728c1fd900a9206afaf874ecd4af14: Status 404 returned error can't find the container with id 080805eb4b018aef29a4a5302846eb0346728c1fd900a9206afaf874ecd4af14
Sep 18 16:53:48.010284 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:48.010249    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm" event={"ID":"c142618b-0b7a-4b05-b41c-dffa1de8cae2","Type":"ContainerStarted","Data":"df236128ef0683f20bca1c7fe9010d4db535fd609624654ac1d17cb43ed83981"}
Sep 18 16:53:48.010689 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:48.010304    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm" event={"ID":"c142618b-0b7a-4b05-b41c-dffa1de8cae2","Type":"ContainerStarted","Data":"080805eb4b018aef29a4a5302846eb0346728c1fd900a9206afaf874ecd4af14"}
Sep 18 16:53:48.010689 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:48.010336    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm"
Sep 18 16:53:48.050879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:48.050824    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm" podStartSLOduration=2.050807469 podStartE2EDuration="2.050807469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:53:48.047080343 +0000 UTC m=+2923.955860707" watchObservedRunningTime="2026-09-18 16:53:48.050807469 +0000 UTC m=+2923.959587820"
Sep 18 16:53:59.018257 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:53:59.018223    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-z9xzm"
Sep 18 16:54:07.071619 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:07.071582    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:54:15.849920 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:15.849881    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:54:24.450454 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:24.450416    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:54:30.149862 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.149780    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"]
Sep 18 16:54:30.150426 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.150407    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c53eeff-c125-49c6-ab26-182dd935f398" containerName="manager"
Sep 18 16:54:30.153825 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.153803    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.158343 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.158322    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-5d102189-kserve-self-signed-certs\""
Sep 18 16:54:30.165807 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.165784    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"]
Sep 18 16:54:30.238677 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.238640    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-model-cache\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.238841 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.238702    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-tmp-dir\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.238841 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.238813    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-home\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.238949 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.238842    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb9qg\" (UniqueName: \"kubernetes.io/projected/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-kube-api-access-fb9qg\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.239007 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.238959    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-dshm\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.239091 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.239054    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-tls-certs\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.339635 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.339596    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-home\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.339845 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.339652    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb9qg\" (UniqueName: \"kubernetes.io/projected/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-kube-api-access-fb9qg\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.339845 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.339710    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-dshm\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.339945 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.339894    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-tls-certs\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.340016 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.339954    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-model-cache\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.340016 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.340001    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-home\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.340154 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.340012    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-tmp-dir\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.340364 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.340344    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-tmp-dir\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.340457 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.340387    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-model-cache\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.341909 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.341892    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-dshm\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.342410 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.342388    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-tls-certs\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.349087 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.349067    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb9qg\" (UniqueName: \"kubernetes.io/projected/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-kube-api-access-fb9qg\") pod \"e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") " pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.463564 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.463540    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:30.738689 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:30.738660    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"]
Sep 18 16:54:30.741364 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:54:30.741326    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8f002e2_d7e2_47b5_baa8_fe74fa38724d.slice/crio-36bc37d0f8de8b1256d17611abc85d4443afd6e639b46a1cab52019d19f5485c WatchSource:0}: Error finding container 36bc37d0f8de8b1256d17611abc85d4443afd6e639b46a1cab52019d19f5485c: Status 404 returned error can't find the container with id 36bc37d0f8de8b1256d17611abc85d4443afd6e639b46a1cab52019d19f5485c
Sep 18 16:54:31.247348 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:31.247312    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp" event={"ID":"c8f002e2-d7e2-47b5-baa8-fe74fa38724d","Type":"ContainerStarted","Data":"c710944b0230c75d984b4eb153ed6f53b9001c79d2c7f5af3e815b6056edfc97"}
Sep 18 16:54:31.247736 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:31.247366    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp" event={"ID":"c8f002e2-d7e2-47b5-baa8-fe74fa38724d","Type":"ContainerStarted","Data":"36bc37d0f8de8b1256d17611abc85d4443afd6e639b46a1cab52019d19f5485c"}
Sep 18 16:54:31.277727 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:31.277678    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp" podStartSLOduration=1.277662266 podStartE2EDuration="1.277662266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:54:31.275602343 +0000 UTC m=+2967.184382695" watchObservedRunningTime="2026-09-18 16:54:31.277662266 +0000 UTC m=+2967.186442627"
Sep 18 16:54:34.661121 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:34.661088    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:54:40.463972 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:40.463939    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:40.464386 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:40.463988    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:40.476170 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:40.476146    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:41.314814 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:41.314783    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:54:47.570907 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.570869    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"]
Sep 18 16:54:47.574117 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.574100    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.582080 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.582060    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-5d102189-kserve-self-signed-certs\""
Sep 18 16:54:47.595969 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.595944    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"]
Sep 18 16:54:47.698246 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.698216    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-model-cache\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.698403 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.698267    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-dshm\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.698403 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.698296    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-home\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.698539 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.698405    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgc9n\" (UniqueName: \"kubernetes.io/projected/db020982-65a2-4cca-a86b-e838fec0196e-kube-api-access-lgc9n\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.698539 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.698456    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-tmp-dir\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.698539 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.698499    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db020982-65a2-4cca-a86b-e838fec0196e-tls-certs\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.799401 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.799366    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-dshm\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.799582 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.799418    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-home\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.799582 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.799466    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgc9n\" (UniqueName: \"kubernetes.io/projected/db020982-65a2-4cca-a86b-e838fec0196e-kube-api-access-lgc9n\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.799582 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.799488    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-tmp-dir\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.799582 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.799510    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db020982-65a2-4cca-a86b-e838fec0196e-tls-certs\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.799582 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.799565    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-model-cache\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.799859 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.799837    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-home\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.799921 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.799899    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-tmp-dir\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.799978 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.799917    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-model-cache\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.801730 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.801710    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-dshm\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.801988 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.801969    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db020982-65a2-4cca-a86b-e838fec0196e-tls-certs\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.813927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.813895    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgc9n\" (UniqueName: \"kubernetes.io/projected/db020982-65a2-4cca-a86b-e838fec0196e-kube-api-access-lgc9n\") pod \"e2e-same-id-b-5d102189-kserve-777588789f-k294x\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") " pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:47.883696 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:47.883634    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:48.196547 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:48.196511    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"]
Sep 18 16:54:48.200430 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:54:48.200402    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb020982_65a2_4cca_a86b_e838fec0196e.slice/crio-fbed46367018fd1395670f4e78770dfa4cc0bfa380662add0171b47a5c887047 WatchSource:0}: Error finding container fbed46367018fd1395670f4e78770dfa4cc0bfa380662add0171b47a5c887047: Status 404 returned error can't find the container with id fbed46367018fd1395670f4e78770dfa4cc0bfa380662add0171b47a5c887047
Sep 18 16:54:48.340783 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:48.340751    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x" event={"ID":"db020982-65a2-4cca-a86b-e838fec0196e","Type":"ContainerStarted","Data":"7f4f2cb40dea087873c00c77c1338ea32fc9c201423601c0bb6c492d244ae005"}
Sep 18 16:54:48.340934 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:48.340798    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x" event={"ID":"db020982-65a2-4cca-a86b-e838fec0196e","Type":"ContainerStarted","Data":"fbed46367018fd1395670f4e78770dfa4cc0bfa380662add0171b47a5c887047"}
Sep 18 16:54:48.368713 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:48.368667    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x" podStartSLOduration=1.3686531290000001 podStartE2EDuration="1.368653129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:54:48.366695829 +0000 UTC m=+2984.275476181" watchObservedRunningTime="2026-09-18 16:54:48.368653129 +0000 UTC m=+2984.277433481"
Sep 18 16:54:51.695402 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:51.695366    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:54:57.884889 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:57.884847    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:57.885410 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:57.884902    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:57.897037 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:57.897002    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:54:58.411035 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:54:58.410986    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:55:05.132802 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:05.132771    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-55f6fc7b86-7xldd_45b59ace-62a2-45ae-9f58-a3d21c8c2ab5/service-ca-controller/1.log"
Sep 18 16:55:05.133491 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:05.133472    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57cdc4b768-7sgr7_8e62eede-ea1b-415e-8549-3df93215459e/service-ca-operator/2.log"
Sep 18 16:55:05.143447 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:05.143426    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-55f6fc7b86-7xldd_45b59ace-62a2-45ae-9f58-a3d21c8c2ab5/service-ca-controller/1.log"
Sep 18 16:55:05.144009 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:05.143993    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57cdc4b768-7sgr7_8e62eede-ea1b-415e-8549-3df93215459e/service-ca-operator/2.log"
Sep 18 16:55:11.520506 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:11.520470    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"]
Sep 18 16:55:12.562170 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:12.562139    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"]
Sep 18 16:55:15.946652 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:15.946616    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:55:26.539589 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.539525    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp" podUID="c8f002e2-d7e2-47b5-baa8-fe74fa38724d" containerName="main" containerID="cri-o://c710944b0230c75d984b4eb153ed6f53b9001c79d2c7f5af3e815b6056edfc97" gracePeriod=45
Sep 18 16:55:26.798229 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.798168    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:55:26.877382 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.877349    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-model-cache\") pod \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") "
Sep 18 16:55:26.877559 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.877401    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-home\") pod \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") "
Sep 18 16:55:26.877559 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.877490    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb9qg\" (UniqueName: \"kubernetes.io/projected/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-kube-api-access-fb9qg\") pod \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") "
Sep 18 16:55:26.877559 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.877553    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-tmp-dir\") pod \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") "
Sep 18 16:55:26.877745 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.877595    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-tls-certs\") pod \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") "
Sep 18 16:55:26.877745 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.877626    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-dshm\") pod \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\" (UID: \"c8f002e2-d7e2-47b5-baa8-fe74fa38724d\") "
Sep 18 16:55:26.877745 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.877645    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8f002e2-d7e2-47b5-baa8-fe74fa38724d" (UID: "c8f002e2-d7e2-47b5-baa8-fe74fa38724d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:55:26.877745 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.877674    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-home" (OuterVolumeSpecName: "home") pod "c8f002e2-d7e2-47b5-baa8-fe74fa38724d" (UID: "c8f002e2-d7e2-47b5-baa8-fe74fa38724d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:55:26.877976 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.877846    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8f002e2-d7e2-47b5-baa8-fe74fa38724d" (UID: "c8f002e2-d7e2-47b5-baa8-fe74fa38724d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:55:26.878044 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.877993    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:55:26.878044 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.878014    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:55:26.878044 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.878043    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:55:26.879879 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.879856    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-dshm" (OuterVolumeSpecName: "dshm") pod "c8f002e2-d7e2-47b5-baa8-fe74fa38724d" (UID: "c8f002e2-d7e2-47b5-baa8-fe74fa38724d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:55:26.879993 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.879917    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8f002e2-d7e2-47b5-baa8-fe74fa38724d" (UID: "c8f002e2-d7e2-47b5-baa8-fe74fa38724d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:55:26.880105 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.880085    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-kube-api-access-fb9qg" (OuterVolumeSpecName: "kube-api-access-fb9qg") pod "c8f002e2-d7e2-47b5-baa8-fe74fa38724d" (UID: "c8f002e2-d7e2-47b5-baa8-fe74fa38724d"). InnerVolumeSpecName "kube-api-access-fb9qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:55:26.979713 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.979682    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:55:26.979713 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.979714    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:55:26.979855 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:26.979728    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb9qg\" (UniqueName: \"kubernetes.io/projected/c8f002e2-d7e2-47b5-baa8-fe74fa38724d-kube-api-access-fb9qg\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:55:27.554308 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.554273    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:55:27.569773 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.569743    2546 generic.go:356] "Generic (PLEG): container finished" podID="c8f002e2-d7e2-47b5-baa8-fe74fa38724d" containerID="c710944b0230c75d984b4eb153ed6f53b9001c79d2c7f5af3e815b6056edfc97" exitCode=0
Sep 18 16:55:27.569914 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.569788    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"
Sep 18 16:55:27.569914 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.569829    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp" event={"ID":"c8f002e2-d7e2-47b5-baa8-fe74fa38724d","Type":"ContainerDied","Data":"c710944b0230c75d984b4eb153ed6f53b9001c79d2c7f5af3e815b6056edfc97"}
Sep 18 16:55:27.569914 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.569884    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp" event={"ID":"c8f002e2-d7e2-47b5-baa8-fe74fa38724d","Type":"ContainerDied","Data":"36bc37d0f8de8b1256d17611abc85d4443afd6e639b46a1cab52019d19f5485c"}
Sep 18 16:55:27.569914 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.569909    2546 scope.go:117] "RemoveContainer" containerID="c710944b0230c75d984b4eb153ed6f53b9001c79d2c7f5af3e815b6056edfc97"
Sep 18 16:55:27.579998 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.579980    2546 scope.go:117] "RemoveContainer" containerID="c710944b0230c75d984b4eb153ed6f53b9001c79d2c7f5af3e815b6056edfc97"
Sep 18 16:55:27.580332 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:55:27.580300    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c710944b0230c75d984b4eb153ed6f53b9001c79d2c7f5af3e815b6056edfc97\": container with ID starting with c710944b0230c75d984b4eb153ed6f53b9001c79d2c7f5af3e815b6056edfc97 not found: ID does not exist" containerID="c710944b0230c75d984b4eb153ed6f53b9001c79d2c7f5af3e815b6056edfc97"
Sep 18 16:55:27.580423 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.580341    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c710944b0230c75d984b4eb153ed6f53b9001c79d2c7f5af3e815b6056edfc97"} err="failed to get container status \"c710944b0230c75d984b4eb153ed6f53b9001c79d2c7f5af3e815b6056edfc97\": rpc error: code = NotFound desc = could not find container \"c710944b0230c75d984b4eb153ed6f53b9001c79d2c7f5af3e815b6056edfc97\": container with ID starting with c710944b0230c75d984b4eb153ed6f53b9001c79d2c7f5af3e815b6056edfc97 not found: ID does not exist"
Sep 18 16:55:27.584397 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.584372    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x" podUID="db020982-65a2-4cca-a86b-e838fec0196e" containerName="main" containerID="cri-o://7f4f2cb40dea087873c00c77c1338ea32fc9c201423601c0bb6c492d244ae005" gracePeriod=45
Sep 18 16:55:27.596817 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.596794    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"]
Sep 18 16:55:27.602493 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.602474    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-5d102189-kserve-5dd5f998c9-qwmtp"]
Sep 18 16:55:27.837673 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.837649    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:55:27.990327 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.990295    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-model-cache\") pod \"db020982-65a2-4cca-a86b-e838fec0196e\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") "
Sep 18 16:55:27.990543 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.990354    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgc9n\" (UniqueName: \"kubernetes.io/projected/db020982-65a2-4cca-a86b-e838fec0196e-kube-api-access-lgc9n\") pod \"db020982-65a2-4cca-a86b-e838fec0196e\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") "
Sep 18 16:55:27.990543 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.990406    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-tmp-dir\") pod \"db020982-65a2-4cca-a86b-e838fec0196e\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") "
Sep 18 16:55:27.990543 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.990439    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-home\") pod \"db020982-65a2-4cca-a86b-e838fec0196e\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") "
Sep 18 16:55:27.990543 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.990465    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-dshm\") pod \"db020982-65a2-4cca-a86b-e838fec0196e\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") "
Sep 18 16:55:27.990543 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.990511    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db020982-65a2-4cca-a86b-e838fec0196e-tls-certs\") pod \"db020982-65a2-4cca-a86b-e838fec0196e\" (UID: \"db020982-65a2-4cca-a86b-e838fec0196e\") "
Sep 18 16:55:27.990814 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.990595    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-model-cache" (OuterVolumeSpecName: "model-cache") pod "db020982-65a2-4cca-a86b-e838fec0196e" (UID: "db020982-65a2-4cca-a86b-e838fec0196e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:55:27.990814 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.990696    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-home" (OuterVolumeSpecName: "home") pod "db020982-65a2-4cca-a86b-e838fec0196e" (UID: "db020982-65a2-4cca-a86b-e838fec0196e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:55:27.990814 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.990710    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db020982-65a2-4cca-a86b-e838fec0196e" (UID: "db020982-65a2-4cca-a86b-e838fec0196e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:55:27.990956 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.990844    2546 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-home\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:55:27.990956 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.990862    2546 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-model-cache\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:55:27.990956 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.990877    2546 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-tmp-dir\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:55:27.992661 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.992628    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db020982-65a2-4cca-a86b-e838fec0196e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db020982-65a2-4cca-a86b-e838fec0196e" (UID: "db020982-65a2-4cca-a86b-e838fec0196e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:55:27.992792 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.992730    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-dshm" (OuterVolumeSpecName: "dshm") pod "db020982-65a2-4cca-a86b-e838fec0196e" (UID: "db020982-65a2-4cca-a86b-e838fec0196e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:55:27.992884 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:27.992866    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db020982-65a2-4cca-a86b-e838fec0196e-kube-api-access-lgc9n" (OuterVolumeSpecName: "kube-api-access-lgc9n") pod "db020982-65a2-4cca-a86b-e838fec0196e" (UID: "db020982-65a2-4cca-a86b-e838fec0196e"). InnerVolumeSpecName "kube-api-access-lgc9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:55:28.092182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:28.092104    2546 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db020982-65a2-4cca-a86b-e838fec0196e-tls-certs\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:55:28.092182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:28.092131    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgc9n\" (UniqueName: \"kubernetes.io/projected/db020982-65a2-4cca-a86b-e838fec0196e-kube-api-access-lgc9n\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:55:28.092182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:28.092141    2546 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db020982-65a2-4cca-a86b-e838fec0196e-dshm\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:55:28.577240 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:28.577206    2546 generic.go:356] "Generic (PLEG): container finished" podID="db020982-65a2-4cca-a86b-e838fec0196e" containerID="7f4f2cb40dea087873c00c77c1338ea32fc9c201423601c0bb6c492d244ae005" exitCode=0
Sep 18 16:55:28.577658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:28.577272    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x" event={"ID":"db020982-65a2-4cca-a86b-e838fec0196e","Type":"ContainerDied","Data":"7f4f2cb40dea087873c00c77c1338ea32fc9c201423601c0bb6c492d244ae005"}
Sep 18 16:55:28.577658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:28.577275    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"
Sep 18 16:55:28.577658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:28.577304    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x" event={"ID":"db020982-65a2-4cca-a86b-e838fec0196e","Type":"ContainerDied","Data":"fbed46367018fd1395670f4e78770dfa4cc0bfa380662add0171b47a5c887047"}
Sep 18 16:55:28.577658 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:28.577321    2546 scope.go:117] "RemoveContainer" containerID="7f4f2cb40dea087873c00c77c1338ea32fc9c201423601c0bb6c492d244ae005"
Sep 18 16:55:28.586521 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:28.586506    2546 scope.go:117] "RemoveContainer" containerID="7f4f2cb40dea087873c00c77c1338ea32fc9c201423601c0bb6c492d244ae005"
Sep 18 16:55:28.586868 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:55:28.586843    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f4f2cb40dea087873c00c77c1338ea32fc9c201423601c0bb6c492d244ae005\": container with ID starting with 7f4f2cb40dea087873c00c77c1338ea32fc9c201423601c0bb6c492d244ae005 not found: ID does not exist" containerID="7f4f2cb40dea087873c00c77c1338ea32fc9c201423601c0bb6c492d244ae005"
Sep 18 16:55:28.586961 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:28.586878    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4f2cb40dea087873c00c77c1338ea32fc9c201423601c0bb6c492d244ae005"} err="failed to get container status \"7f4f2cb40dea087873c00c77c1338ea32fc9c201423601c0bb6c492d244ae005\": rpc error: code = NotFound desc = could not find container \"7f4f2cb40dea087873c00c77c1338ea32fc9c201423601c0bb6c492d244ae005\": container with ID starting with 7f4f2cb40dea087873c00c77c1338ea32fc9c201423601c0bb6c492d244ae005 not found: ID does not exist"
Sep 18 16:55:28.602991 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:28.602964    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8f002e2-d7e2-47b5-baa8-fe74fa38724d" path="/var/lib/kubelet/pods/c8f002e2-d7e2-47b5-baa8-fe74fa38724d/volumes"
Sep 18 16:55:28.603580 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:28.603561    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"]
Sep 18 16:55:28.609523 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:28.609501    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-5d102189-kserve-777588789f-k294x"]
Sep 18 16:55:30.602714 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:30.602682    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db020982-65a2-4cca-a86b-e838fec0196e" path="/var/lib/kubelet/pods/db020982-65a2-4cca-a86b-e838fec0196e/volumes"
Sep 18 16:55:37.569462 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:37.569428    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:55:46.053563 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:46.053526    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:55:56.353249 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:56.353213    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:55:59.442555 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.442477    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67dfb4599-462gk"]
Sep 18 16:55:59.442942 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.442880    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db020982-65a2-4cca-a86b-e838fec0196e" containerName="main"
Sep 18 16:55:59.442942 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.442894    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="db020982-65a2-4cca-a86b-e838fec0196e" containerName="main"
Sep 18 16:55:59.442942 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.442912    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8f002e2-d7e2-47b5-baa8-fe74fa38724d" containerName="main"
Sep 18 16:55:59.442942 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.442918    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8f002e2-d7e2-47b5-baa8-fe74fa38724d" containerName="main"
Sep 18 16:55:59.443118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.442996    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="db020982-65a2-4cca-a86b-e838fec0196e" containerName="main"
Sep 18 16:55:59.443118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.443005    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8f002e2-d7e2-47b5-baa8-fe74fa38724d" containerName="main"
Sep 18 16:55:59.446143 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.446121    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.453267 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.453249    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e444cec2-c3a7-4604-a8c3-e5b0befe253d-trusted-ca-bundle\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.453371 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.453277    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxlx5\" (UniqueName: \"kubernetes.io/projected/e444cec2-c3a7-4604-a8c3-e5b0befe253d-kube-api-access-qxlx5\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.453371 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.453299    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e444cec2-c3a7-4604-a8c3-e5b0befe253d-console-oauth-config\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.453371 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.453338    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e444cec2-c3a7-4604-a8c3-e5b0befe253d-console-config\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.453498 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.453374    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e444cec2-c3a7-4604-a8c3-e5b0befe253d-service-ca\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.453498 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.453407    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e444cec2-c3a7-4604-a8c3-e5b0befe253d-oauth-serving-cert\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.453498 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.453431    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e444cec2-c3a7-4604-a8c3-e5b0befe253d-console-serving-cert\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.465137 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.465113    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67dfb4599-462gk"]
Sep 18 16:55:59.554877 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.554848    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e444cec2-c3a7-4604-a8c3-e5b0befe253d-trusted-ca-bundle\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.554997 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.554880    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxlx5\" (UniqueName: \"kubernetes.io/projected/e444cec2-c3a7-4604-a8c3-e5b0befe253d-kube-api-access-qxlx5\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.554997 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.554905    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e444cec2-c3a7-4604-a8c3-e5b0befe253d-console-oauth-config\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.555136 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.555041    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e444cec2-c3a7-4604-a8c3-e5b0befe253d-console-config\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.555136 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.555074    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e444cec2-c3a7-4604-a8c3-e5b0befe253d-service-ca\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.555136 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.555102    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e444cec2-c3a7-4604-a8c3-e5b0befe253d-oauth-serving-cert\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.555136 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.555126    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e444cec2-c3a7-4604-a8c3-e5b0befe253d-console-serving-cert\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.555958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.555931    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e444cec2-c3a7-4604-a8c3-e5b0befe253d-service-ca\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.556097 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.555932    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e444cec2-c3a7-4604-a8c3-e5b0befe253d-oauth-serving-cert\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.556097 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.555985    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e444cec2-c3a7-4604-a8c3-e5b0befe253d-console-config\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.556097 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.555984    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e444cec2-c3a7-4604-a8c3-e5b0befe253d-trusted-ca-bundle\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.557680 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.557657    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e444cec2-c3a7-4604-a8c3-e5b0befe253d-console-oauth-config\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.557783 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.557666    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e444cec2-c3a7-4604-a8c3-e5b0befe253d-console-serving-cert\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.565376 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.565356    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxlx5\" (UniqueName: \"kubernetes.io/projected/e444cec2-c3a7-4604-a8c3-e5b0befe253d-kube-api-access-qxlx5\") pod \"console-67dfb4599-462gk\" (UID: \"e444cec2-c3a7-4604-a8c3-e5b0befe253d\") " pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:55:59.755602 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:55:59.755525    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:56:00.044970 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:00.044940    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67dfb4599-462gk"]
Sep 18 16:56:00.045976 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:56:00.045944    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode444cec2_c3a7_4604_a8c3_e5b0befe253d.slice/crio-bf7dedc1a3afca5f204e7645bc440d528b8f3e886e831edde4dc1162d1fac07c WatchSource:0}: Error finding container bf7dedc1a3afca5f204e7645bc440d528b8f3e886e831edde4dc1162d1fac07c: Status 404 returned error can't find the container with id bf7dedc1a3afca5f204e7645bc440d528b8f3e886e831edde4dc1162d1fac07c
Sep 18 16:56:00.762267 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:00.762221    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67dfb4599-462gk" event={"ID":"e444cec2-c3a7-4604-a8c3-e5b0befe253d","Type":"ContainerStarted","Data":"e72845e963d842572295808a6acf54c9564850ef36afec8cccc9397e70a8c919"}
Sep 18 16:56:00.762725 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:00.762320    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67dfb4599-462gk" event={"ID":"e444cec2-c3a7-4604-a8c3-e5b0befe253d","Type":"ContainerStarted","Data":"bf7dedc1a3afca5f204e7645bc440d528b8f3e886e831edde4dc1162d1fac07c"}
Sep 18 16:56:00.791309 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:00.791257    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67dfb4599-462gk" podStartSLOduration=1.7912424740000001 podStartE2EDuration="1.791242474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:56:00.789496233 +0000 UTC m=+3056.698276611" watchObservedRunningTime="2026-09-18 16:56:00.791242474 +0000 UTC m=+3056.700022825"
Sep 18 16:56:07.956903 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:07.956872    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:56:09.756607 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:09.756576    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:56:09.756972 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:09.756637    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:56:09.761102 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:09.761050    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:56:09.816891 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:09.816871    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67dfb4599-462gk"
Sep 18 16:56:09.894317 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:09.894286    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-595cbfdbf7-cx7g5"]
Sep 18 16:56:15.655923 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:15.655892    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jw9lb"]
Sep 18 16:56:18.075943 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:18.075918    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-766f75f9d7-btp4s_fa3e43bc-9fbf-4748-a4a6-b497c0556912/authorino/0.log"
Sep 18 16:56:23.556002 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:23.555970    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pxtkv_f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7/manager/6.log"
Sep 18 16:56:23.671839 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:23.671814    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8bfgv_d9b87cf9-1252-4eeb-bd58-194c81ea4a94/manager/3.log"
Sep 18 16:56:23.908181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:23.908113    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-g6r2f_84711704-c750-4a14-bace-15805d228697/server/0.log"
Sep 18 16:56:24.022583 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:24.022559    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-klwkg_4ce26bbe-bd40-4653-a16f-2b0c1c87604b/manager/3.log"
Sep 18 16:56:24.264859 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:24.264837    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws_eb9ee00b-5814-4425-a7e3-4062562574cc/copy-manifests/0.log"
Sep 18 16:56:24.272667 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:24.272643    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws_eb9ee00b-5814-4425-a7e3-4062562574cc/manager/5.log"
Sep 18 16:56:25.222714 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:25.222687    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829150-djk5t_c13899d7-4176-4980-88d9-93e3641f2950/cleanup/0.log"
Sep 18 16:56:25.459370 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:25.459343    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829150-jsbcr_41ccf59d-3206-4435-9e2c-8766851cdf4f/cleanup/0.log"
Sep 18 16:56:25.583438 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:25.583376    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-a-w3-a75117-29829165-sddfk_a7cf6fba-8b83-4edb-9c6d-8e9fada753f1/cleanup/0.log"
Sep 18 16:56:25.811964 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:25.811941    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-e2e-shared-b-w3-11725e-29829165-kxvjt_f00fd175-7121-417a-8eb0-badc33e0ae8f/cleanup/0.log"
Sep 18 16:56:26.724570 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:26.724541    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-766f75f9d7-btp4s_fa3e43bc-9fbf-4748-a4a6-b497c0556912/authorino/0.log"
Sep 18 16:56:26.855455 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:26.855416    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-t7kmf_f7e45976-cc30-418c-96f4-70e0c667572e/manager/3.log"
Sep 18 16:56:27.214732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:27.214704    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-z9xzm_c142618b-0b7a-4b05-b41c-dffa1de8cae2/manager/0.log"
Sep 18 16:56:27.334339 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:27.334315    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw9lb_7c162150-470f-4f52-bf7a-9cb35c68bb9c/limitador/0.log"
Sep 18 16:56:27.452964 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:27.452940    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-dbsdv_61d67412-30af-4fb0-adc0-5251f88b7a41/manager/3.log"
Sep 18 16:56:27.824788 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:27.824761    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m_a9ca2260-8b26-43f3-9373-cf4ee2e25d9a/istio-proxy/0.log"
Sep 18 16:56:27.938319 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:27.938297    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8_822e99a6-4dbb-43d9-b068-7237dd24b74c/istio-proxy/1.log"
Sep 18 16:56:28.062358 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:28.062333    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76_1b4328b9-39c4-4fb3-9527-c4aa699774fc/istio-proxy/1.log"
Sep 18 16:56:28.433905 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:28.433856    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f97b6d9fd-tl7dn_16e823a1-2033-49b7-80be-18435d2aead9/kube-auth-proxy/0.log"
Sep 18 16:56:28.550743 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:28.550717    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-chbcf_43a91800-e401-43ef-9427-597e42651fe1/istio-proxy/1.log"
Sep 18 16:56:29.377132 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:29.377102    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-847f74c87d-nnkct_fdf89ce0-2abf-4a51-a332-8c9c01121c6a/router/0.log"
Sep 18 16:56:30.392532 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:30.392502    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-5mk7z_b10fd0c8-5fe6-4adb-8fb5-773c5dd80ee2/main/0.log"
Sep 18 16:56:34.914911 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:34.914861    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-595cbfdbf7-cx7g5" podUID="6143688a-e1f7-4af0-915f-649305788fbe" containerName="console" containerID="cri-o://561c1aef3ab9d54bc7aa141804bdc4f2191087166f258dbb44a9a670ef421e5f" gracePeriod=15
Sep 18 16:56:35.155883 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.155864    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:56:35.264037 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.264002    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6143688a-e1f7-4af0-915f-649305788fbe-console-serving-cert\") pod \"6143688a-e1f7-4af0-915f-649305788fbe\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") "
Sep 18 16:56:35.264181 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.264103    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6143688a-e1f7-4af0-915f-649305788fbe-console-oauth-config\") pod \"6143688a-e1f7-4af0-915f-649305788fbe\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") "
Sep 18 16:56:35.264223 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.264185    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-oauth-serving-cert\") pod \"6143688a-e1f7-4af0-915f-649305788fbe\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") "
Sep 18 16:56:35.264266 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.264234    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd7v9\" (UniqueName: \"kubernetes.io/projected/6143688a-e1f7-4af0-915f-649305788fbe-kube-api-access-bd7v9\") pod \"6143688a-e1f7-4af0-915f-649305788fbe\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") "
Sep 18 16:56:35.264312 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.264299    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-service-ca\") pod \"6143688a-e1f7-4af0-915f-649305788fbe\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") "
Sep 18 16:56:35.264366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.264324    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-trusted-ca-bundle\") pod \"6143688a-e1f7-4af0-915f-649305788fbe\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") "
Sep 18 16:56:35.264366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.264341    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-console-config\") pod \"6143688a-e1f7-4af0-915f-649305788fbe\" (UID: \"6143688a-e1f7-4af0-915f-649305788fbe\") "
Sep 18 16:56:35.264644 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.264549    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6143688a-e1f7-4af0-915f-649305788fbe" (UID: "6143688a-e1f7-4af0-915f-649305788fbe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:56:35.264802 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.264672    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-service-ca" (OuterVolumeSpecName: "service-ca") pod "6143688a-e1f7-4af0-915f-649305788fbe" (UID: "6143688a-e1f7-4af0-915f-649305788fbe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:56:35.264914 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.264797    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6143688a-e1f7-4af0-915f-649305788fbe" (UID: "6143688a-e1f7-4af0-915f-649305788fbe"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:56:35.264914 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.264880    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-console-config" (OuterVolumeSpecName: "console-config") pod "6143688a-e1f7-4af0-915f-649305788fbe" (UID: "6143688a-e1f7-4af0-915f-649305788fbe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:56:35.266339 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.266321    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6143688a-e1f7-4af0-915f-649305788fbe-kube-api-access-bd7v9" (OuterVolumeSpecName: "kube-api-access-bd7v9") pod "6143688a-e1f7-4af0-915f-649305788fbe" (UID: "6143688a-e1f7-4af0-915f-649305788fbe"). InnerVolumeSpecName "kube-api-access-bd7v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:56:35.266582 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.266566    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6143688a-e1f7-4af0-915f-649305788fbe-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6143688a-e1f7-4af0-915f-649305788fbe" (UID: "6143688a-e1f7-4af0-915f-649305788fbe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:56:35.266653 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.266576    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6143688a-e1f7-4af0-915f-649305788fbe-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6143688a-e1f7-4af0-915f-649305788fbe" (UID: "6143688a-e1f7-4af0-915f-649305788fbe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:56:35.365691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.365670    2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-service-ca\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:56:35.365691 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.365690    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-trusted-ca-bundle\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:56:35.365834 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.365699    2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-console-config\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:56:35.365834 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.365707    2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6143688a-e1f7-4af0-915f-649305788fbe-console-serving-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:56:35.365834 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.365716    2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6143688a-e1f7-4af0-915f-649305788fbe-console-oauth-config\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:56:35.365834 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.365724    2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6143688a-e1f7-4af0-915f-649305788fbe-oauth-serving-cert\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:56:35.365834 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.365732    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd7v9\" (UniqueName: \"kubernetes.io/projected/6143688a-e1f7-4af0-915f-649305788fbe-kube-api-access-bd7v9\") on node \"ip-10-0-135-253.ec2.internal\" DevicePath \"\""
Sep 18 16:56:35.521010 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.520945    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7rjqm/must-gather-wwvnc"]
Sep 18 16:56:35.521374 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.521357    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6143688a-e1f7-4af0-915f-649305788fbe" containerName="console"
Sep 18 16:56:35.521451 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.521377    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6143688a-e1f7-4af0-915f-649305788fbe" containerName="console"
Sep 18 16:56:35.521530 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.521518    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6143688a-e1f7-4af0-915f-649305788fbe" containerName="console"
Sep 18 16:56:35.524880 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.524861    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7rjqm/must-gather-wwvnc"
Sep 18 16:56:35.532435 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.532416    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7rjqm\"/\"default-dockercfg-8knlp\""
Sep 18 16:56:35.532535 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.532436    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7rjqm\"/\"openshift-service-ca.crt\""
Sep 18 16:56:35.532669 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.532654    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7rjqm\"/\"kube-root-ca.crt\""
Sep 18 16:56:35.542361 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.542338    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7rjqm/must-gather-wwvnc"]
Sep 18 16:56:35.667373 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.667348    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3-must-gather-output\") pod \"must-gather-wwvnc\" (UID: \"6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3\") " pod="openshift-must-gather-7rjqm/must-gather-wwvnc"
Sep 18 16:56:35.667554 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.667449    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl2b8\" (UniqueName: \"kubernetes.io/projected/6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3-kube-api-access-jl2b8\") pod \"must-gather-wwvnc\" (UID: \"6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3\") " pod="openshift-must-gather-7rjqm/must-gather-wwvnc"
Sep 18 16:56:35.768695 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.768665    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl2b8\" (UniqueName: \"kubernetes.io/projected/6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3-kube-api-access-jl2b8\") pod \"must-gather-wwvnc\" (UID: \"6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3\") " pod="openshift-must-gather-7rjqm/must-gather-wwvnc"
Sep 18 16:56:35.768795 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.768714    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3-must-gather-output\") pod \"must-gather-wwvnc\" (UID: \"6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3\") " pod="openshift-must-gather-7rjqm/must-gather-wwvnc"
Sep 18 16:56:35.769073 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.769053    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3-must-gather-output\") pod \"must-gather-wwvnc\" (UID: \"6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3\") " pod="openshift-must-gather-7rjqm/must-gather-wwvnc"
Sep 18 16:56:35.794236 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.794178    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl2b8\" (UniqueName: \"kubernetes.io/projected/6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3-kube-api-access-jl2b8\") pod \"must-gather-wwvnc\" (UID: \"6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3\") " pod="openshift-must-gather-7rjqm/must-gather-wwvnc"
Sep 18 16:56:35.834176 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.834155    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7rjqm/must-gather-wwvnc"
Sep 18 16:56:35.958475 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.958448    2546 generic.go:356] "Generic (PLEG): container finished" podID="6143688a-e1f7-4af0-915f-649305788fbe" containerID="561c1aef3ab9d54bc7aa141804bdc4f2191087166f258dbb44a9a670ef421e5f" exitCode=0
Sep 18 16:56:35.958884 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.958488    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595cbfdbf7-cx7g5"
Sep 18 16:56:35.958884 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.958488    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595cbfdbf7-cx7g5" event={"ID":"6143688a-e1f7-4af0-915f-649305788fbe","Type":"ContainerDied","Data":"561c1aef3ab9d54bc7aa141804bdc4f2191087166f258dbb44a9a670ef421e5f"}
Sep 18 16:56:35.958884 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.958553    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595cbfdbf7-cx7g5" event={"ID":"6143688a-e1f7-4af0-915f-649305788fbe","Type":"ContainerDied","Data":"3c559423cebfe9ab3a1fb4ff7a022dfd5b3e28673617899f101e2909b8a0fd70"}
Sep 18 16:56:35.958884 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.958583    2546 scope.go:117] "RemoveContainer" containerID="561c1aef3ab9d54bc7aa141804bdc4f2191087166f258dbb44a9a670ef421e5f"
Sep 18 16:56:35.967372 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.967357    2546 scope.go:117] "RemoveContainer" containerID="561c1aef3ab9d54bc7aa141804bdc4f2191087166f258dbb44a9a670ef421e5f"
Sep 18 16:56:35.967656 ip-10-0-135-253 kubenswrapper[2546]: E0918 16:56:35.967627    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561c1aef3ab9d54bc7aa141804bdc4f2191087166f258dbb44a9a670ef421e5f\": container with ID starting with 561c1aef3ab9d54bc7aa141804bdc4f2191087166f258dbb44a9a670ef421e5f not found: ID does not exist" containerID="561c1aef3ab9d54bc7aa141804bdc4f2191087166f258dbb44a9a670ef421e5f"
Sep 18 16:56:35.967746 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.967663    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561c1aef3ab9d54bc7aa141804bdc4f2191087166f258dbb44a9a670ef421e5f"} err="failed to get container status \"561c1aef3ab9d54bc7aa141804bdc4f2191087166f258dbb44a9a670ef421e5f\": rpc error: code = NotFound desc = could not find container \"561c1aef3ab9d54bc7aa141804bdc4f2191087166f258dbb44a9a670ef421e5f\": container with ID starting with 561c1aef3ab9d54bc7aa141804bdc4f2191087166f258dbb44a9a670ef421e5f not found: ID does not exist"
Sep 18 16:56:35.987348 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.987328    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-595cbfdbf7-cx7g5"]
Sep 18 16:56:35.995525 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:35.995506    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-595cbfdbf7-cx7g5"]
Sep 18 16:56:36.111979 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:36.111958    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7rjqm/must-gather-wwvnc"]
Sep 18 16:56:36.112946 ip-10-0-135-253 kubenswrapper[2546]: W0918 16:56:36.112928    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c6925e2_5bd8_4e4d_ae9d_1093a6631ae3.slice/crio-81b8de6bf91e02da5db9a077ed4658eecebb271898dfa25f9db893fd899ccdc8 WatchSource:0}: Error finding container 81b8de6bf91e02da5db9a077ed4658eecebb271898dfa25f9db893fd899ccdc8: Status 404 returned error can't find the container with id 81b8de6bf91e02da5db9a077ed4658eecebb271898dfa25f9db893fd899ccdc8
Sep 18 16:56:36.114802 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:36.114785    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:56:36.604208 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:36.604178    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6143688a-e1f7-4af0-915f-649305788fbe" path="/var/lib/kubelet/pods/6143688a-e1f7-4af0-915f-649305788fbe/volumes"
Sep 18 16:56:36.969366 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:36.969330    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7rjqm/must-gather-wwvnc" event={"ID":"6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3","Type":"ContainerStarted","Data":"e58a742034bc13d4779dc5b282cb523f330ed4dd888c607b11e269c20a26d51c"}
Sep 18 16:56:36.969775 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:36.969384    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7rjqm/must-gather-wwvnc" event={"ID":"6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3","Type":"ContainerStarted","Data":"81b8de6bf91e02da5db9a077ed4658eecebb271898dfa25f9db893fd899ccdc8"}
Sep 18 16:56:37.979436 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:37.978627    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7rjqm/must-gather-wwvnc" event={"ID":"6c6925e2-5bd8-4e4d-ae9d-1093a6631ae3","Type":"ContainerStarted","Data":"ebae2aab5bbee45aaf5330efe78455111e8bf4e96050393efa7a3dd525e6d2cc"}
Sep 18 16:56:38.023086 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:38.023014    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7rjqm/must-gather-wwvnc" podStartSLOduration=2.290954791 podStartE2EDuration="3.022992543s" totalImagesPullingTime="732.037752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:56:38.021541648 +0000 UTC m=+3093.930321999" watchObservedRunningTime="2026-09-18 16:56:38.022992543 +0000 UTC m=+3093.931772893"
Sep 18 16:56:38.666900 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:38.666858    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pcmn7_7cfa77c5-aa51-41ef-9673-c23c39835582/global-pull-secret-syncer/0.log"
Sep 18 16:56:38.809898 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:38.809824    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-js9rq_43e780cc-b6d9-456a-9dbc-eedb54a237a2/konnectivity-agent/0.log"
Sep 18 16:56:38.936574 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:38.936506    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-253.ec2.internal_93cd48393ba29a5c2da55b21c3ff83e3/haproxy/0.log"
Sep 18 16:56:43.409116 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:43.407678    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-766f75f9d7-btp4s_fa3e43bc-9fbf-4748-a4a6-b497c0556912/authorino/0.log"
Sep 18 16:56:43.464648 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:43.464386    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-t7kmf_f7e45976-cc30-418c-96f4-70e0c667572e/manager/3.log"
Sep 18 16:56:43.464920 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:43.464770    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-t7kmf_f7e45976-cc30-418c-96f4-70e0c667572e/manager/2.log"
Sep 18 16:56:43.592905 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:43.592827    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-z9xzm_c142618b-0b7a-4b05-b41c-dffa1de8cae2/manager/0.log"
Sep 18 16:56:43.627433 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:43.627401    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jw9lb_7c162150-470f-4f52-bf7a-9cb35c68bb9c/limitador/0.log"
Sep 18 16:56:43.675060 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:43.675010    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-dbsdv_61d67412-30af-4fb0-adc0-5251f88b7a41/manager/2.log"
Sep 18 16:56:43.682760 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:43.682728    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-dbsdv_61d67412-30af-4fb0-adc0-5251f88b7a41/manager/3.log"
Sep 18 16:56:45.636231 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:45.636156    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-vwnjj_9ec0ffc7-1d21-4f0d-be23-507156419b8b/cluster-monitoring-operator/0.log"
Sep 18 16:56:45.758201 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:45.758161    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-798fb7c859-58tmk_c1ba48c5-1444-4082-aa66-d33328cf5882/metrics-server/0.log"
Sep 18 16:56:45.981871 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:45.981834    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gbrnd_3a9469a9-e638-4bfb-ae8e-09cff76a4e1c/node-exporter/0.log"
Sep 18 16:56:46.013308 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:46.013268    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gbrnd_3a9469a9-e638-4bfb-ae8e-09cff76a4e1c/kube-rbac-proxy/0.log"
Sep 18 16:56:46.046430 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:46.046398    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gbrnd_3a9469a9-e638-4bfb-ae8e-09cff76a4e1c/init-textfile/0.log"
Sep 18 16:56:48.275872 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.275842    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-rpw7p_6dd55250-53ec-4e73-a50c-0fba9472f458/networking-console-plugin/0.log"
Sep 18 16:56:48.278724 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.278695    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"]
Sep 18 16:56:48.282266 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.282236    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.303128 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.303103    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"]
Sep 18 16:56:48.336410 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.336357    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-lib-modules\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.336732 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.336709    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-sys\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.336884 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.336866    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-podres\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.337094 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.337079    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-proc\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.337260 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.337242    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bnfg\" (UniqueName: \"kubernetes.io/projected/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-kube-api-access-6bnfg\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.438539 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.438504    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-lib-modules\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.438835 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.438770    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-lib-modules\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.439047 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.438826    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-sys\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.439221 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.439202    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-podres\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.439375 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.439349    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-podres\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.439466 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.438887    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-sys\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.439632 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.439611    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-proc\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.439830 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.439787    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-proc\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.439958 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.439828    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnfg\" (UniqueName: \"kubernetes.io/projected/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-kube-api-access-6bnfg\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.452220 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.452199    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnfg\" (UniqueName: \"kubernetes.io/projected/4507d9ba-ad6a-42d9-8adb-a1f945fd58b8-kube-api-access-6bnfg\") pod \"perf-node-gather-daemonset-6fbnf\" (UID: \"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8\") " pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.596087 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.595985    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:48.890461 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:48.887347    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"]
Sep 18 16:56:49.070558 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:49.070517    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf" event={"ID":"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8","Type":"ContainerStarted","Data":"fdb55e74a3638d684b355e58db4fa82de211116985d06324e9e923dc5336bb38"}
Sep 18 16:56:49.070742 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:49.070583    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf" event={"ID":"4507d9ba-ad6a-42d9-8adb-a1f945fd58b8","Type":"ContainerStarted","Data":"1632c66f55db7b78c55be5f8b9e560a74b8fe01e88b55db6f284bde40802c941"}
Sep 18 16:56:49.070742 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:49.070676    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:49.093675 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:49.093614    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf" podStartSLOduration=1.093595144 podStartE2EDuration="1.093595144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:56:49.0918967 +0000 UTC m=+3105.000677065" watchObservedRunningTime="2026-09-18 16:56:49.093595144 +0000 UTC m=+3105.002375497"
Sep 18 16:56:49.616479 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:49.616450    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67dfb4599-462gk_e444cec2-c3a7-4604-a8c3-e5b0befe253d/console/0.log"
Sep 18 16:56:51.481166 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:51.481140    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c7lb2_ad2fa51d-439b-425a-af0e-05a48b372205/dns/0.log"
Sep 18 16:56:51.517605 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:51.517577    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c7lb2_ad2fa51d-439b-425a-af0e-05a48b372205/kube-rbac-proxy/0.log"
Sep 18 16:56:51.800146 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:51.800079    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2bl6_2023cf66-25c9-441a-b1b9-6892cf0b2746/dns-node-resolver/0.log"
Sep 18 16:56:52.465356 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:52.465329    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-667f675df5-l9bfb_f835fe9f-0f6a-4ba6-b277-3897ab30a5f6/registry/0.log"
Sep 18 16:56:52.477804 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:52.477758    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-667f675df5-l9bfb_f835fe9f-0f6a-4ba6-b277-3897ab30a5f6/registry/1.log"
Sep 18 16:56:52.538182 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:52.538142    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6gkbl_49ffc9f3-7c5f-48bc-8217-838ac6651979/node-ca/0.log"
Sep 18 16:56:53.576538 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:53.576509    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m_a9ca2260-8b26-43f3-9373-cf4ee2e25d9a/istio-proxy/0.log"
Sep 18 16:56:53.606851 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:53.606821    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8_822e99a6-4dbb-43d9-b068-7237dd24b74c/istio-proxy/0.log"
Sep 18 16:56:53.607086 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:53.606901    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8_822e99a6-4dbb-43d9-b068-7237dd24b74c/istio-proxy/1.log"
Sep 18 16:56:53.634754 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:53.634703    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76_1b4328b9-39c4-4fb3-9527-c4aa699774fc/istio-proxy/0.log"
Sep 18 16:56:53.634754 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:53.634727    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76_1b4328b9-39c4-4fb3-9527-c4aa699774fc/istio-proxy/1.log"
Sep 18 16:56:54.073664 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:54.073637    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f97b6d9fd-tl7dn_16e823a1-2033-49b7-80be-18435d2aead9/kube-auth-proxy/0.log"
Sep 18 16:56:54.111530 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:54.111499    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-chbcf_43a91800-e401-43ef-9427-597e42651fe1/istio-proxy/0.log"
Sep 18 16:56:54.114343 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:54.114321    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-chbcf_43a91800-e401-43ef-9427-597e42651fe1/istio-proxy/1.log"
Sep 18 16:56:54.346611 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:54.346538    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-847f74c87d-nnkct_fdf89ce0-2abf-4a51-a332-8c9c01121c6a/router/0.log"
Sep 18 16:56:55.091230 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:55.091204    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7rjqm/perf-node-gather-daemonset-6fbnf"
Sep 18 16:56:55.387074 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:55.386982    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r7pf8_2de82e27-28f6-4ee2-bf84-74b92409a9b8/serve-healthcheck-canary/0.log"
Sep 18 16:56:55.938588 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:55.938553    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-69shv_a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b/kube-rbac-proxy/0.log"
Sep 18 16:56:55.966524 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:55.966502    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-69shv_a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b/exporter/0.log"
Sep 18 16:56:56.002094 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:56.002070    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-69shv_a83c2bf2-11a5-4eb2-b38e-d7e94cc4fe4b/extractor/0.log"
Sep 18 16:56:58.355426 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:58.355393    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m_a9ca2260-8b26-43f3-9373-cf4ee2e25d9a/istio-proxy/0.log"
Sep 18 16:56:58.382460 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:58.382428    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8_822e99a6-4dbb-43d9-b068-7237dd24b74c/istio-proxy/0.log"
Sep 18 16:56:58.382627 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:58.382611    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8_822e99a6-4dbb-43d9-b068-7237dd24b74c/istio-proxy/1.log"
Sep 18 16:56:58.396617 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:58.396593    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76_1b4328b9-39c4-4fb3-9527-c4aa699774fc/istio-proxy/0.log"
Sep 18 16:56:58.396727 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:58.396706    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76_1b4328b9-39c4-4fb3-9527-c4aa699774fc/istio-proxy/1.log"
Sep 18 16:56:58.812883 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:58.812850    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f97b6d9fd-tl7dn_16e823a1-2033-49b7-80be-18435d2aead9/kube-auth-proxy/0.log"
Sep 18 16:56:58.832612 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:58.832585    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pxtkv_f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7/manager/5.log"
Sep 18 16:56:58.833458 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:58.833429    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-pxtkv_f8bdde87-b8d3-4fa1-81db-9cd1668bbfb7/manager/6.log"
Sep 18 16:56:58.835891 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:58.835872    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-chbcf_43a91800-e401-43ef-9427-597e42651fe1/istio-proxy/0.log"
Sep 18 16:56:58.838549 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:58.838533    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-chbcf_43a91800-e401-43ef-9427-597e42651fe1/istio-proxy/1.log"
Sep 18 16:56:58.865960 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:58.865928    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8bfgv_d9b87cf9-1252-4eeb-bd58-194c81ea4a94/manager/2.log"
Sep 18 16:56:58.896809 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:58.896789    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8bfgv_d9b87cf9-1252-4eeb-bd58-194c81ea4a94/manager/3.log"
Sep 18 16:56:58.988105 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:58.988084    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-g6r2f_84711704-c750-4a14-bace-15805d228697/server/0.log"
Sep 18 16:56:59.015827 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:59.015803    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-847f74c87d-nnkct_fdf89ce0-2abf-4a51-a332-8c9c01121c6a/router/0.log"
Sep 18 16:56:59.030128 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:59.030108    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-klwkg_4ce26bbe-bd40-4653-a16f-2b0c1c87604b/manager/2.log"
Sep 18 16:56:59.033630 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:59.033611    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-klwkg_4ce26bbe-bd40-4653-a16f-2b0c1c87604b/manager/3.log"
Sep 18 16:56:59.124451 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:59.124375    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws_eb9ee00b-5814-4425-a7e3-4062562574cc/manager/4.log"
Sep 18 16:56:59.124631 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:59.124617    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws_eb9ee00b-5814-4425-a7e3-4062562574cc/manager/5.log"
Sep 18 16:56:59.150738 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:56:59.150713    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cdc5fd7cb-6dmws_eb9ee00b-5814-4425-a7e3-4062562574cc/copy-manifests/0.log"
Sep 18 16:57:00.595347 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:57:00.595315    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cfsl4m_a9ca2260-8b26-43f3-9373-cf4ee2e25d9a/istio-proxy/0.log"
Sep 18 16:57:00.618927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:57:00.618897    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8_822e99a6-4dbb-43d9-b068-7237dd24b74c/istio-proxy/0.log"
Sep 18 16:57:00.619118 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:57:00.618928    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-a-w3-a75117-openshift-default-6cb5f5cc6-ctxq8_822e99a6-4dbb-43d9-b068-7237dd24b74c/istio-proxy/1.log"
Sep 18 16:57:00.634786 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:57:00.634766    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76_1b4328b9-39c4-4fb3-9527-c4aa699774fc/istio-proxy/0.log"
Sep 18 16:57:00.634927 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:57:00.634913    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_e2e-shared-b-w3-11725e-openshift-default-77f9ff9cbd-s6s76_1b4328b9-39c4-4fb3-9527-c4aa699774fc/istio-proxy/1.log"
Sep 18 16:57:01.088967 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:57:01.088944    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f97b6d9fd-tl7dn_16e823a1-2033-49b7-80be-18435d2aead9/kube-auth-proxy/0.log"
Sep 18 16:57:01.114158 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:57:01.114130    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-chbcf_43a91800-e401-43ef-9427-597e42651fe1/istio-proxy/0.log"
Sep 18 16:57:01.117682 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:57:01.117661    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-chbcf_43a91800-e401-43ef-9427-597e42651fe1/istio-proxy/1.log"
Sep 18 16:57:01.274858 ip-10-0-135-253 kubenswrapper[2546]: I0918 16:57:01.274831    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-847f74c87d-nnkct_fdf89ce0-2abf-4a51-a332-8c9c01121c6a/router/0.log"