Sep 10 12:30:24.453892 ip-10-0-133-150 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 12:30:24.860729 ip-10-0-133-150 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 12:30:24.860729 ip-10-0-133-150 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 12:30:24.860729 ip-10-0-133-150 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 12:30:24.860729 ip-10-0-133-150 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 12:30:24.860729 ip-10-0-133-150 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 12:30:24.862385 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.862295    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 12:30:24.866230 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866213    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:30:24.866230 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866229    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866233    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866239    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866244    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866247    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866250    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866253    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866256    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866258    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866261    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866263    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866266    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866269    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866271    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866274    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866278    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866281    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866284    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866287    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:30:24.866299 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866290    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866293    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866296    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866298    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866301    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866304    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866307    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866310    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866313    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866319    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866322    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866325    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866327    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866330    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866332    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866335    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866337    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866340    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866342    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866345    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:30:24.866759 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866347    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866350    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866352    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866355    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866357    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866360    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866363    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866365    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866368    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866372    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866374    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866377    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866379    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866382    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866385    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866388    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866391    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866393    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866396    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866398    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:30:24.867234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866401    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866403    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866406    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866408    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866411    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866414    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866416    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866419    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866421    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866424    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866428    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866432    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866434    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866437    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866440    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866442    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866445    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866447    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866450    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:30:24.867740 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866453    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866456    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866459    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866462    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866464    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866467    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866469    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866472    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866475    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866477    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866480    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866483    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866485    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866488    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866490    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866493    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866495    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866498    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866500    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:30:24.868209 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.866503    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867227    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867234    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867237    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867240    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867243    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867245    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867248    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867251    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867255    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867259    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867262    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867266    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867269    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867272    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867275    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867278    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867281    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867283    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867286    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:30:24.868679 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867288    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867291    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867294    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867296    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867299    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867302    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867304    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867307    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867309    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867312    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867314    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867316    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867320    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867323    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867326    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867328    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867330    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867333    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867336    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867339    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:30:24.869187 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867341    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867344    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867346    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867348    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867351    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867354    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867357    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867359    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867362    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867365    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867367    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867369    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867373    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867375    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867379    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867383    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867386    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867389    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867392    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:30:24.869800 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867395    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867398    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867402    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867404    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867407    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867410    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867412    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867415    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867417    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867420    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867422    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867425    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867427    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867430    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867432    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867435    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867437    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867439    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867442    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:30:24.870275 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867444    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867447    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867451    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867454    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867457    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867460    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867463    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867466    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867468    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867471    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867474    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867477    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867479    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867482    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867485    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867487    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867490    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867492    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867495    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:30:24.870772 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867498    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867500    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867503    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867919    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867925    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867928    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867931    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867934    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867936    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867941    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867945    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867948    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867950    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867954    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867957    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867959    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867963    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867965    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867968    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:30:24.871257 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867971    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867974    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867976    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867979    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867982    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867984    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867987    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867992    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867996    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.867999    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868002    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868005    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868007    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868010    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868012    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868015    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868017    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868020    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868023    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868025    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:30:24.871763 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868028    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868030    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868033    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868035    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868038    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868040    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868043    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868045    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868048    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868050    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868053    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868056    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868059    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868062    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868064    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868067    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868070    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868072    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868075    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868078    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:30:24.872262 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868081    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868084    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868087    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868090    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868093    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868095    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868098    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868101    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868103    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868106    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868108    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868111    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868113    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868116    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868119    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868121    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868124    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868126    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868129    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868131    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:30:24.872766 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868134    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868136    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868139    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868142    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868146    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868149    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868152    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868155    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868158    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868161    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868163    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868167    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868170    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868173    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868175    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868178    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868180    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868183    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868185    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:30:24.873286 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868188    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868190    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868193    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868196    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868286    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868337    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868345    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868350    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868354    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868357    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868361    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868366    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868369    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868372    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868375    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868378    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868381    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868388    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868391    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868394    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868397    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868400    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868404    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868407    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 12:30:24.873775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868411    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868414    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868418    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868422    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868425    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868428    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868431    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868434    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868437    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868440    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868443    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868446    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868450    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868453    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868456    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868459    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868462    2563 flags.go:64] FLAG: --enable-server="true"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868465    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868471    2563 flags.go:64] FLAG: --event-burst="100"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868474    2563 flags.go:64] FLAG: --event-qps="50"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868477    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868480    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868483    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868487    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 12:30:24.874352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868490    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868493    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868497    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868500    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868503    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868505    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868508    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868511    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868515    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868518    2563 flags.go:64] FLAG: --feature-gates=""
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868522    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868525    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868528    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868531    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868534    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868537    2563 flags.go:64] FLAG: --help="false"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868540    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-133-150.ec2.internal"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868543    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868546    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868549    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868552    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868556    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868558    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 12:30:24.875011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868561    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868564    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868567    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868571    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868574    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868577    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868580    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868583    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868586    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868601    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868604    2563 flags.go:64] FLAG: --lock-file=""
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868607    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868610    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868613    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868618    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868621    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868624    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868629    2563 flags.go:64] FLAG: --logging-format="text"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868631    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868635    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868638    2563 flags.go:64] FLAG: --manifest-url=""
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868641    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868645    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868648    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868652    2563 flags.go:64] FLAG: --max-pods="110"
Sep 10 12:30:24.875610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868655    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868658    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868661    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868665    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868668    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868671    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868674    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868682    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868685    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868688    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868693    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868696    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868702    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868705    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868708    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868711    2563 flags.go:64] FLAG: --port="10250"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868714    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868718    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f11a1963b1cc5d67"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868721    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868724    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868727    2563 flags.go:64] FLAG: --register-node="true"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868730    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868734    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868737    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 10 12:30:24.876259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868741    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868744    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868748    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868751    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868754    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868757    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868760    2563 flags.go:64] FLAG: --runonce="false"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868763    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868766    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868769    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868772    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868775    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868778    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868781    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868785    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868788    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868791    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868793    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868796    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868801    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868804    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868807    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868812    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868815    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868818    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 12:30:24.876866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868821    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868824    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868827    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868830    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868836    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868839    2563 flags.go:64] FLAG: --v="2"
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868843    2563 flags.go:64] FLAG: --version="false"
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868849    2563 flags.go:64] FLAG: --vmodule=""
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868853    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.868856    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868990    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868994    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868997    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.868999    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869002    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869005    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869008    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869011    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869013    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869016    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869018    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869021    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:30:24.877477 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869023    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869026    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869028    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869032    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869038    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869040    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869043    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869046    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869048    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869051    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869053    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869056    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869060    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869063    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869067    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869070    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869074    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869078    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869080    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:30:24.878038 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869083    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869086    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869088    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869091    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869093    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869096    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869099    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869101    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869103    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869106    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869108    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869111    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869114    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869116    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869119    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869121    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869124    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869128    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869131    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869133    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:30:24.878537 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869136    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869138    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869141    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869143    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869146    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869148    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869151    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869153    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869155    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869158    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869162    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869165    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869167    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869170    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869172    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869175    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869177    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869180    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869182    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869184    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:30:24.879112 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869187    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869190    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869193    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869195    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869199    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869201    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869204    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869207    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869209    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869213    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869217    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869220    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869224    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869227    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869229    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869232    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869235    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869237    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869240    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:30:24.879631 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869243    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869245    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869248    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869252    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869255    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869257    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869260    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869262    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.869265    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.870154    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.877369    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.877385    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.878195    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.878206    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 12:30:24.880109 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878265    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878270    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878273    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878277    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878280    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878283    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878286    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878289    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878291    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878294    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878297    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878300    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878303    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878305    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878308    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878311    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878314    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878317    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878319    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878322    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:30:24.880462 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878325    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878327    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878330    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878332    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878335    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878337    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878340    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878343    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878346    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878350    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878353    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878358    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878361    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878364    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878366    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878369    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878372    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878374    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878377    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878380    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:30:24.880965 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878382    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878385    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878387    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878390    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878392    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878395    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878398    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878400    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878403    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878405    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878408    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878410    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878413    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878415    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878418    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878421    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878423    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878426    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878429    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878431    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:30:24.881507 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878435    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878438    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878441    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878444    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878446    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878449    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878451    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878454    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878456    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878461    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878464    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878467    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878470    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878473    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878476    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878478    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878482    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878485    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878489    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:30:24.882060 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878492    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878495    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878497    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878500    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878502    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878505    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878508    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878510    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878513    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878515    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878518    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878520    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878523    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878526    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878529    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878532    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878535    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878538    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878541    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:30:24.882699 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878544    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.878549    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878679    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878685    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878688    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878691    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878694    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878697    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878699    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878702    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878705    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878707    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878710    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878713    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:30:24.883197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878716    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878719    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878721    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878724    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878726    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878729    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878731    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878734    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878736    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878739    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878741    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878744    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878747    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878749    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878753    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878757    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878761    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878765    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878768    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:30:24.883552 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878771    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878774    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878777    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878780    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878782    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878785    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878788    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878791    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878794    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878796    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878799    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878801    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878804    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878806    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878809    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878812    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878815    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878817    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878820    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:30:24.884054 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878822    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878825    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878827    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878830    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878832    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878835    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878838    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878841    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878843    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878846    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878849    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878852    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878855    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878858    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878860    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878863    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878866    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878869    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878871    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878874    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:30:24.884609 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878876    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878879    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878881    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878884    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878886    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878889    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878891    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878894    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878896    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878899    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878902    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878905    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878907    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878910    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878912    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878915    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878917    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878920    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878922    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878925    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878928    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:30:24.885235 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878930    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:30:24.886125 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878933    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:30:24.886125 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878935    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:30:24.886125 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878938    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:30:24.886125 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878941    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:30:24.886125 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878943    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:30:24.886125 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878946    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:30:24.886125 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:24.878949    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:30:24.886125 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.878953    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 12:30:24.886125 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.879059    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 12:30:24.886125 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.881066    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 12:30:24.886125 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.882023    2563 server.go:1019] "Starting client certificate rotation"
Sep 10 12:30:24.886125 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.882138    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 12:30:24.886125 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.882193    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 12:30:24.886650 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.885026    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 12:30:24.886650 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.886202    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 12:30:24.898627 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.898607    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 10 12:30:24.901517 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.901498    2563 log.go:25] "Validated CRI v1 image API"
Sep 10 12:30:24.902446 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.902429    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 12:30:24.905220 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.905195    2563 fs.go:135] Filesystem UUIDs: map[1266856f-ac6e-4ba8-9768-62c892ba7590:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d750c29a-f1d3-4958-b747-75a74d74077b:/dev/nvme0n1p4]
Sep 10 12:30:24.905281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.905220    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 12:30:24.911022 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.910910    2563 manager.go:216] Machine: {Timestamp:2026-09-10 12:30:24.909557787 +0000 UTC m=+0.335429589 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104369 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29da429def958a0a4c0a0326405c7d SystemUUID:ec29da42-9def-958a-0a4c-0a0326405c7d BootID:d956795a-5956-4940-aeb2-16cb4b060cb6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f6:e4:90:16:d7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f6:e4:90:16:d7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f2:c0:c9:48:1d:1c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 12:30:24.911022 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.911019    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 12:30:24.911130 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.911100    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 12:30:24.914891 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.914827    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 12:30:24.915042 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.914895    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-150.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 12:30:24.916382 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.916369    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 12:30:24.916382 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.916384    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 12:30:24.916470 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.916392    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 12:30:24.917141 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.917130    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 12:30:24.918011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.917998    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 12:30:24.919960 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.919942    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 12:30:24.921131 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.921119    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 12:30:24.921248 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.921238    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 12:30:24.921287 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.921269    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 12:30:24.921287 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.921284    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 12:30:24.921343 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.921302    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 12:30:24.921343 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.921311    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 10 12:30:24.921343 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.921323    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 12:30:24.921737 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.921725    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 12:30:24.921778 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.921746    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 12:30:24.923864 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.923847    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 12:30:24.926122 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.926107    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 12:30:24.926187 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.926137    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 12:30:24.927245 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.927234    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 12:30:24.927281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.927250    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 12:30:24.927281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.927256    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 12:30:24.927281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.927261    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 12:30:24.927281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.927270    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 12:30:24.927281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.927276    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 12:30:24.927417 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.927289    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 12:30:24.927417 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.927295    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 12:30:24.927417 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.927302    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 12:30:24.927417 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.927307    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 12:30:24.927417 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.927325    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 12:30:24.927417 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.927330    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 12:30:24.928053 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.928043    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 12:30:24.928053 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.928053    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 12:30:24.930022 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.930006    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-150.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 12:30:24.930639 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:24.930621    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-150.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 12:30:24.930696 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:24.930621    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 12:30:24.932323 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.932311    2563 server.go:1268] "Started kubelet"
Sep 10 12:30:24.932416 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.932392    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 12:30:24.932497 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.932451    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 12:30:24.932543 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.932524    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 12:30:24.933870 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.933852    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 12:30:24.934765 ip-10-0-133-150 systemd[1]: Started Kubernetes Kubelet.
Sep 10 12:30:24.939695 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:24.939674    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 12:30:24.940828 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.940813    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 12:30:24.943956 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.943935    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 12:30:24.945164 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.945146    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 12:30:24.945319 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:24.945298    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:24.945902 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.945884    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 12:30:24.945902 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.945885    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 12:30:24.946014 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.945912    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 12:30:24.946014 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.945932    2563 factory.go:55] Registering systemd factory
Sep 10 12:30:24.946014 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.945946    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 10 12:30:24.946014 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.946012    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 12:30:24.946180 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.946020    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 12:30:24.946180 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.946160    2563 factory.go:153] Registering CRI-O factory
Sep 10 12:30:24.946180 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.946178    2563 factory.go:223] Registration of the crio container factory successfully
Sep 10 12:30:24.946297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.946234    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 12:30:24.946297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.946248    2563 factory.go:103] Registering Raw factory
Sep 10 12:30:24.946297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.946263    2563 manager.go:1196] Started watching for new ooms in manager
Sep 10 12:30:24.946681 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.946660    2563 manager.go:349] Starting recovery of all containers
Sep 10 12:30:24.947510 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:24.947481    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 12:30:24.947622 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:24.947541    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-150.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 12:30:24.948295 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.948275    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7pfrv"
Sep 10 12:30:24.948941 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:24.947661    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-150.ec2.internal.18d3f58426d64203  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-150.ec2.internal,UID:ip-10-0-133-150.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-150.ec2.internal,},FirstTimestamp:2026-09-10 12:30:24.932282883 +0000 UTC m=+0.358154685,LastTimestamp:2026-09-10 12:30:24.932282883 +0000 UTC m=+0.358154685,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-150.ec2.internal,}"
Sep 10 12:30:24.954788 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.954606    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7pfrv"
Sep 10 12:30:24.955957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.955937    2563 manager.go:354] Recovery completed
Sep 10 12:30:24.960870 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.960857    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:24.962862 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.962847    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:24.962930 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.962877    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:24.962930 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.962889    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:24.963423 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.963408    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 12:30:24.963423 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.963422    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 12:30:24.963520 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.963437    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 12:30:24.963520 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.963416    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:24.964250 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.964230    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:24.964321 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.964260    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:24.964321 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.964278    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:24.964656 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.964642    2563 policy_none.go:49] "None policy: Start"
Sep 10 12:30:24.964728 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.964663    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 12:30:24.964728 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.964677    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 12:30:24.964963 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:24.964862    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-150.ec2.internal.18d3f58428a8d7e4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-150.ec2.internal,UID:ip-10-0-133-150.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-150.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-150.ec2.internal,},FirstTimestamp:2026-09-10 12:30:24.962861028 +0000 UTC m=+0.388732831,LastTimestamp:2026-09-10 12:30:24.962861028 +0000 UTC m=+0.388732831,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-150.ec2.internal,}"
Sep 10 12:30:24.965895 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.965883    2563 policy_none.go:47] "Start"
Sep 10 12:30:24.993328 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.993303    2563 manager.go:337] "Starting Device Plugin manager"
Sep 10 12:30:25.009608 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:24.993351    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 12:30:25.009608 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.993365    2563 server.go:87] "Starting device plugin registration server"
Sep 10 12:30:25.009608 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.993753    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 12:30:25.009608 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.993764    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 12:30:25.009608 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.993870    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 12:30:25.009608 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.994107    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 12:30:25.009608 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:24.994118    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 12:30:25.009608 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:24.994722    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 12:30:25.009608 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:24.994791    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:25.010255 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.010206    2563 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2599 into the system container \"/system.slice\": <nil>"
Sep 10 12:30:25.094291 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.094243    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:25.095340 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.095314    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:25.095464 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.095358    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:25.095464 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.095374    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:25.095464 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.095405    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.103332 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.103315    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.103393 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:25.103341    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-150.ec2.internal\": node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:25.120704 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:25.120615    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:25.195564 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.195533    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 12:30:25.196770 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.196746    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 12:30:25.196883 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.196795    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 12:30:25.196883 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.196816    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 12:30:25.196981 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:25.196862    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 12:30:25.200805 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.200786    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:30:25.221028 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:25.221011    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:25.297562 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.297527    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-150.ec2.internal"]
Sep 10 12:30:25.297703 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.297636    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:25.298694 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.298677    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:25.298785 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.298704    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:25.298785 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.298719    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:25.300298 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.300285    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:25.300425 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.300412    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.300467 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.300441    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:25.301058 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.301041    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:25.301154 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.301068    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:25.301154 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.301083    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:25.301154 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.301096    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:25.301154 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.301121    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:25.301154 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.301135    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:25.302557 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.302541    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.302666 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.302570    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:30:25.303216 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.303197    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:30:25.303305 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.303226    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:30:25.303305 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.303236    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:30:25.321311 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:25.321287    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:25.330465 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:25.330438    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-150.ec2.internal\" not found" node="ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.333974 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:25.333959    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-150.ec2.internal\" not found" node="ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.348466 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.348440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3a4ab1e3c1464e6c152bc80398459b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal\" (UID: \"f3a4ab1e3c1464e6c152bc80398459b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.348604 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.348466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3a4ab1e3c1464e6c152bc80398459b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal\" (UID: \"f3a4ab1e3c1464e6c152bc80398459b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.348604 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.348492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87b3b84e20b1589782e644bb9f5ff7c5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-150.ec2.internal\" (UID: \"87b3b84e20b1589782e644bb9f5ff7c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.422225 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:25.422152    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:25.449542 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.449517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3a4ab1e3c1464e6c152bc80398459b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal\" (UID: \"f3a4ab1e3c1464e6c152bc80398459b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.449643 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.449550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3a4ab1e3c1464e6c152bc80398459b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal\" (UID: \"f3a4ab1e3c1464e6c152bc80398459b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.449643 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.449568    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87b3b84e20b1589782e644bb9f5ff7c5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-150.ec2.internal\" (UID: \"87b3b84e20b1589782e644bb9f5ff7c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.449643 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.449637    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f3a4ab1e3c1464e6c152bc80398459b4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal\" (UID: \"f3a4ab1e3c1464e6c152bc80398459b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.449746 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.449655    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87b3b84e20b1589782e644bb9f5ff7c5-config\") pod \"kube-apiserver-proxy-ip-10-0-133-150.ec2.internal\" (UID: \"87b3b84e20b1589782e644bb9f5ff7c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:25.449746 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:25.449656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3a4ab1e3c1464e6c152bc80398459b4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal\" (UID: \"f3a4ab1e3c1464e6c152bc80398459b4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:27.023669 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:27.023634    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:27.124228 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:27.124191    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:27.134377 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.134352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:27.139634 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.139616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:27.225191 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:27.225109    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:27.325624 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:27.325596    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:27.328261 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.328245    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:30:27.382632 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.382599    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 12:30:27.383049 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.382745    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 12:30:27.383049 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.382785    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 12:30:27.426125 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:27.426083    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:27.460445 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.460409    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 12:25:26 +0000 UTC" deadline="2028-02-10 19:23:14.277448322 +0000 UTC"
Sep 10 12:30:27.460566 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.460450    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12438h52m46.817000755s"
Sep 10 12:30:27.526810 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:27.526740    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:27.627220 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:27.627193    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:27.727814 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:27.727766    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:27.757883 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.757865    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:30:27.799563 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.799470    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:30:27.828490 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:27.828463    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-150.ec2.internal\" not found"
Sep 10 12:30:27.840926 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.839920    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:30:27.846824 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.846802    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:27.857991 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.857967    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 12:30:27.858994 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.858975    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-150.ec2.internal"
Sep 10 12:30:27.859522 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:27.859496    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b3b84e20b1589782e644bb9f5ff7c5.slice/crio-e97f7a1f696159aef7fe9eaa39d963ae73935a98cfeab0d331d2eae3921396ba WatchSource:0}: Error finding container e97f7a1f696159aef7fe9eaa39d963ae73935a98cfeab0d331d2eae3921396ba: Status 404 returned error can't find the container with id e97f7a1f696159aef7fe9eaa39d963ae73935a98cfeab0d331d2eae3921396ba
Sep 10 12:30:27.867386 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:27.867366    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 12:30:28.423660 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.423625    2563 apiserver.go:52] "Watching apiserver"
Sep 10 12:30:28.430442 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.430409    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 12:30:28.432598 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.432564    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-q6wmj","kube-system/kube-apiserver-proxy-ip-10-0-133-150.ec2.internal","openshift-cluster-node-tuning-operator/tuned-q9g7f","openshift-image-registry/node-ca-n5899","openshift-multus/multus-484p6","openshift-multus/multus-additional-cni-plugins-ntbv6","openshift-ovn-kubernetes/ovnkube-node-dtnsr","kube-system/global-pull-secret-syncer-jfpm9","kube-system/konnectivity-agent-sc5dh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh","openshift-dns/node-resolver-j7fz9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal","openshift-multus/network-metrics-daemon-vj4n4","openshift-network-diagnostics/network-check-target-jz9tr"]
Sep 10 12:30:28.434571 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.434524    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.436370 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.436041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:28.436370 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:28.436112    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:28.437801 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.437538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 12:30:28.437801 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.437575    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qr8pc\""
Sep 10 12:30:28.437801 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.437644    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 12:30:28.437995 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.437866    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.438632 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.438269    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 12:30:28.438632 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.438442    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 12:30:28.438632 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.438462    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 12:30:28.438827 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.438803    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 12:30:28.439349 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.439330    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n5899"
Sep 10 12:30:28.440144 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.439980    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 12:30:28.440144 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.440081    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ll9dp\""
Sep 10 12:30:28.440144 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.440096    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:30:28.441693 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.441106    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.441693 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.441634    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 12:30:28.441693 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.441650    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 12:30:28.442062 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.441712    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 12:30:28.442062 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.441750    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7t76v\""
Sep 10 12:30:28.442727 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.442709    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.443372 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.443333    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 12:30:28.443456 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.443421    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 12:30:28.443631 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.443613    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 12:30:28.443710 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.443618    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gsnkx\""
Sep 10 12:30:28.443773 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.443724    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 12:30:28.444325 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.444251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sc5dh"
Sep 10 12:30:28.445847 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.445775    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 12:30:28.445930 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.445853    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 12:30:28.449400 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.449345    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q6wmj"
Sep 10 12:30:28.450048 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.449847    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 12:30:28.450048 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.449905    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zgr22\""
Sep 10 12:30:28.450048 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.449908    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 12:30:28.450048 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.449907    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 12:30:28.451329 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.450313    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wp9zq\""
Sep 10 12:30:28.452910 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.452336    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 12:30:28.452910 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.452441    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 12:30:28.452910 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.452336    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bjsg6\""
Sep 10 12:30:28.452910 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.452734    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:28.452910 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.452737    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:30:28.452910 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:28.452853    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:28.453981 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.453938    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.455595 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.455574    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7fz9"
Sep 10 12:30:28.457001 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.456216    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h5g56\""
Sep 10 12:30:28.457001 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.456430    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 12:30:28.457001 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.456615    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 12:30:28.457001 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.456823    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 12:30:28.457700 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.457685    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 12:30:28.458051 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.458034    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-k5h6l\""
Sep 10 12:30:28.458344 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.458325    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 12:30:28.459364 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-run-systemd\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.459449 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-cni-netd\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.459449 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459402    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:28.459577 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459403    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77f37552-1d26-4529-bdb1-88d5f2b3af43-cni-binary-copy\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.459577 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:28.459455    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:28.459577 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/688eff1f-bb3f-4223-9721-9c8898af37e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.459577 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.459577 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-kubernetes\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.459842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/688eff1f-bb3f-4223-9721-9c8898af37e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.459842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-slash\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.459842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-node-log\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.459842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-log-socket\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.459842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-run\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.459842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-sys\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.459842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459739    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1e745f6-42ce-443e-8555-1572dbad7050-host\") pod \"node-ca-n5899\" (UID: \"c1e745f6-42ce-443e-8555-1572dbad7050\") " pod="openshift-image-registry/node-ca-n5899"
Sep 10 12:30:28.459842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459767    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ad10c96-2142-41c0-836a-7622bf8767bb-host-slash\") pod \"iptables-alerter-q6wmj\" (UID: \"5ad10c96-2142-41c0-836a-7622bf8767bb\") " pod="openshift-network-operator/iptables-alerter-q6wmj"
Sep 10 12:30:28.459842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-modprobe-d\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.459842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-multus-cni-dir\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.459842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-run-netns\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-var-lib-cni-bin\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-hostroot\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-ovnkube-config\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-sysconfig\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-var-lib-kubelet\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.459979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-multus-socket-dir-parent\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtpvz\" (UniqueName: \"kubernetes.io/projected/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-kube-api-access-dtpvz\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1e745f6-42ce-443e-8555-1572dbad7050-serviceca\") pod \"node-ca-n5899\" (UID: \"c1e745f6-42ce-443e-8555-1572dbad7050\") " pod="openshift-image-registry/node-ca-n5899"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460049    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-system-cni-dir\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-systemd-units\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-run-ovn-kubernetes\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-sysctl-conf\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77f37552-1d26-4529-bdb1-88d5f2b3af43-multus-daemon-config\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-run-openvswitch\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-ovnkube-script-lib\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460262    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lkk9\" (UniqueName: \"kubernetes.io/projected/c1e745f6-42ce-443e-8555-1572dbad7050-kube-api-access-2lkk9\") pod \"node-ca-n5899\" (UID: \"c1e745f6-42ce-443e-8555-1572dbad7050\") " pod="openshift-image-registry/node-ca-n5899"
Sep 10 12:30:28.460359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-tmp\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-etc-kubernetes\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wnpl\" (UniqueName: \"kubernetes.io/projected/688eff1f-bb3f-4223-9721-9c8898af37e8-kube-api-access-2wnpl\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-kubelet\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-env-overrides\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/851a86dd-ec09-426e-a81e-71c143dacb4f-konnectivity-ca\") pod \"konnectivity-agent-sc5dh\" (UID: \"851a86dd-ec09-426e-a81e-71c143dacb4f\") " pod="kube-system/konnectivity-agent-sc5dh"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2t2s\" (UniqueName: \"kubernetes.io/projected/a04476a3-6d83-40ff-879c-c9a5b620ef44-kube-api-access-m2t2s\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-ovn-node-metrics-cert\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460516    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-tuned\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/688eff1f-bb3f-4223-9721-9c8898af37e8-cnibin\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/688eff1f-bb3f-4223-9721-9c8898af37e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460610    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-run-netns\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460639    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5ad10c96-2142-41c0-836a-7622bf8767bb-iptables-alerter-script\") pod \"iptables-alerter-q6wmj\" (UID: \"5ad10c96-2142-41c0-836a-7622bf8767bb\") " pod="openshift-network-operator/iptables-alerter-q6wmj"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-sysctl-d\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-var-lib-kubelet\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460719    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rhmq\" (UniqueName: \"kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq\") pod \"network-check-target-jz9tr\" (UID: \"911473b1-59c6-4916-a2d6-2dca448752a2\") " pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:28.461195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-cni-bin\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460783    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-run-k8s-cni-cncf-io\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-var-lib-cni-multus\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-run-multus-certs\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-etc-openvswitch\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-systemd\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-host\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-cnibin\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6kwz\" (UniqueName: \"kubernetes.io/projected/77f37552-1d26-4529-bdb1-88d5f2b3af43-kube-api-access-m6kwz\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.460991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-var-lib-openvswitch\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.461014    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkcnw\" (UniqueName: \"kubernetes.io/projected/5ad10c96-2142-41c0-836a-7622bf8767bb-kube-api-access-rkcnw\") pod \"iptables-alerter-q6wmj\" (UID: \"5ad10c96-2142-41c0-836a-7622bf8767bb\") " pod="openshift-network-operator/iptables-alerter-q6wmj"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.461043    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-multus-conf-dir\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.461068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/688eff1f-bb3f-4223-9721-9c8898af37e8-system-cni-dir\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.461091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-run-ovn\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.461114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/851a86dd-ec09-426e-a81e-71c143dacb4f-agent-certs\") pod \"konnectivity-agent-sc5dh\" (UID: \"851a86dd-ec09-426e-a81e-71c143dacb4f\") " pod="kube-system/konnectivity-agent-sc5dh"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.461145    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-lib-modules\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.461957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.461168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgk9k\" (UniqueName: \"kubernetes.io/projected/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-kube-api-access-tgk9k\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.462976 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.461200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/688eff1f-bb3f-4223-9721-9c8898af37e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.464588 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.464402    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 12:30:28.480562 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.480527    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tlg9z"
Sep 10 12:30:28.488186 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.488164    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tlg9z"
Sep 10 12:30:28.547746 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.547720    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 12:30:28.562272 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-multus-conf-dir\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.562447 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/688eff1f-bb3f-4223-9721-9c8898af37e8-system-cni-dir\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.562447 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-run-ovn\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.562447 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/851a86dd-ec09-426e-a81e-71c143dacb4f-agent-certs\") pod \"konnectivity-agent-sc5dh\" (UID: \"851a86dd-ec09-426e-a81e-71c143dacb4f\") " pod="kube-system/konnectivity-agent-sc5dh"
Sep 10 12:30:28.562447 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/688eff1f-bb3f-4223-9721-9c8898af37e8-system-cni-dir\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.562447 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-lib-modules\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.562719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-run-ovn\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.562719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-multus-conf-dir\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.562719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgk9k\" (UniqueName: \"kubernetes.io/projected/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-kube-api-access-tgk9k\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.562719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/688eff1f-bb3f-4223-9721-9c8898af37e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.562719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-sys-fs\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.562719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcnn7\" (UniqueName: \"kubernetes.io/projected/a6e7308e-da70-4a2b-845a-51057e2f1f94-kube-api-access-tcnn7\") pod \"node-resolver-j7fz9\" (UID: \"a6e7308e-da70-4a2b-845a-51057e2f1f94\") " pod="openshift-dns/node-resolver-j7fz9"
Sep 10 12:30:28.562719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562682    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-run-systemd\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.562719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-cni-netd\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.562719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-registration-dir\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.563064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77f37552-1d26-4529-bdb1-88d5f2b3af43-cni-binary-copy\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.563064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562754    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/688eff1f-bb3f-4223-9721-9c8898af37e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.563064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-lib-modules\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.563064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.563064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-run-systemd\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.563064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.563064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-cni-netd\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.563064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562880    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6e7308e-da70-4a2b-845a-51057e2f1f94-hosts-file\") pod \"node-resolver-j7fz9\" (UID: \"a6e7308e-da70-4a2b-845a-51057e2f1f94\") " pod="openshift-dns/node-resolver-j7fz9"
Sep 10 12:30:28.563064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-kubernetes\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.563064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.562964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/688eff1f-bb3f-4223-9721-9c8898af37e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.563064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-slash\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.563064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-node-log\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.563064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-log-socket\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lnjw\" (UniqueName: \"kubernetes.io/projected/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-kube-api-access-2lnjw\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6e7308e-da70-4a2b-845a-51057e2f1f94-tmp-dir\") pod \"node-resolver-j7fz9\" (UID: \"a6e7308e-da70-4a2b-845a-51057e2f1f94\") " pod="openshift-dns/node-resolver-j7fz9"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-run\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-sys\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1e745f6-42ce-443e-8555-1572dbad7050-host\") pod \"node-ca-n5899\" (UID: \"c1e745f6-42ce-443e-8555-1572dbad7050\") " pod="openshift-image-registry/node-ca-n5899"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ad10c96-2142-41c0-836a-7622bf8767bb-host-slash\") pod \"iptables-alerter-q6wmj\" (UID: \"5ad10c96-2142-41c0-836a-7622bf8767bb\") " pod="openshift-network-operator/iptables-alerter-q6wmj"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563256    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-modprobe-d\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-multus-cni-dir\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-run-netns\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563372    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-var-lib-cni-bin\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77f37552-1d26-4529-bdb1-88d5f2b3af43-cni-binary-copy\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563385    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/688eff1f-bb3f-4223-9721-9c8898af37e8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-hostroot\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-ovnkube-config\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-slash\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-sysconfig\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563455    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-node-log\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.563662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-log-socket\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563224    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-sysconfig\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-multus-cni-dir\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-var-lib-kubelet\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-kubernetes\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-multus-socket-dir-parent\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-var-lib-cni-bin\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-run\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-hostroot\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtpvz\" (UniqueName: \"kubernetes.io/projected/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-kube-api-access-dtpvz\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ad10c96-2142-41c0-836a-7622bf8767bb-host-slash\") pod \"iptables-alerter-q6wmj\" (UID: \"5ad10c96-2142-41c0-836a-7622bf8767bb\") " pod="openshift-network-operator/iptables-alerter-q6wmj"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-run-netns\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564013    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/688eff1f-bb3f-4223-9721-9c8898af37e8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.563902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/688eff1f-bb3f-4223-9721-9c8898af37e8-cni-binary-copy\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-ovnkube-config\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-modprobe-d\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-var-lib-kubelet\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.564455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1e745f6-42ce-443e-8555-1572dbad7050-host\") pod \"node-ca-n5899\" (UID: \"c1e745f6-42ce-443e-8555-1572dbad7050\") " pod="openshift-image-registry/node-ca-n5899"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1e745f6-42ce-443e-8555-1572dbad7050-serviceca\") pod \"node-ca-n5899\" (UID: \"c1e745f6-42ce-443e-8555-1572dbad7050\") " pod="openshift-image-registry/node-ca-n5899"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-sys\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-device-dir\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4be22799-8b7d-4e18-867f-34a3185af15a-kubelet-config\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-system-cni-dir\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-systemd-units\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-run-ovn-kubernetes\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564376    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-system-cni-dir\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-sysctl-conf\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564422    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-run-ovn-kubernetes\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564425    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77f37552-1d26-4529-bdb1-88d5f2b3af43-multus-daemon-config\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-run-openvswitch\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-ovnkube-script-lib\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564507    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lkk9\" (UniqueName: \"kubernetes.io/projected/c1e745f6-42ce-443e-8555-1572dbad7050-kube-api-access-2lkk9\") pod \"node-ca-n5899\" (UID: \"c1e745f6-42ce-443e-8555-1572dbad7050\") " pod="openshift-image-registry/node-ca-n5899"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-systemd-units\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.565297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-tmp\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-etc-kubernetes\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wnpl\" (UniqueName: \"kubernetes.io/projected/688eff1f-bb3f-4223-9721-9c8898af37e8-kube-api-access-2wnpl\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-kubelet\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-run-openvswitch\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-env-overrides\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-etc-kubernetes\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564673    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/851a86dd-ec09-426e-a81e-71c143dacb4f-konnectivity-ca\") pod \"konnectivity-agent-sc5dh\" (UID: \"851a86dd-ec09-426e-a81e-71c143dacb4f\") " pod="kube-system/konnectivity-agent-sc5dh"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2t2s\" (UniqueName: \"kubernetes.io/projected/a04476a3-6d83-40ff-879c-c9a5b620ef44-kube-api-access-m2t2s\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564779    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-ovn-node-metrics-cert\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-tuned\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/688eff1f-bb3f-4223-9721-9c8898af37e8-cnibin\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/688eff1f-bb3f-4223-9721-9c8898af37e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-run-netns\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564914    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-socket-dir\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77f37552-1d26-4529-bdb1-88d5f2b3af43-multus-daemon-config\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-etc-selinux\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.566035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564965    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4be22799-8b7d-4e18-867f-34a3185af15a-dbus\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5ad10c96-2142-41c0-836a-7622bf8767bb-iptables-alerter-script\") pod \"iptables-alerter-q6wmj\" (UID: \"5ad10c96-2142-41c0-836a-7622bf8767bb\") " pod="openshift-network-operator/iptables-alerter-q6wmj"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-sysctl-d\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565025    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-ovnkube-script-lib\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-var-lib-kubelet\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-sysctl-conf\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhmq\" (UniqueName: \"kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq\") pod \"network-check-target-jz9tr\" (UID: \"911473b1-59c6-4916-a2d6-2dca448752a2\") " pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-run-netns\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-cni-bin\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/851a86dd-ec09-426e-a81e-71c143dacb4f-konnectivity-ca\") pod \"konnectivity-agent-sc5dh\" (UID: \"851a86dd-ec09-426e-a81e-71c143dacb4f\") " pod="kube-system/konnectivity-agent-sc5dh"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565197    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-run-k8s-cni-cncf-io\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-var-lib-cni-multus\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-run-multus-certs\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-etc-openvswitch\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.564596    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1e745f6-42ce-443e-8555-1572dbad7050-serviceca\") pod \"node-ca-n5899\" (UID: \"c1e745f6-42ce-443e-8555-1572dbad7050\") " pod="openshift-image-registry/node-ca-n5899"
Sep 10 12:30:28.566879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-systemd\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-host\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-cnibin\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-kubelet\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6kwz\" (UniqueName: \"kubernetes.io/projected/77f37552-1d26-4529-bdb1-88d5f2b3af43-kube-api-access-m6kwz\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-var-lib-openvswitch\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/688eff1f-bb3f-4223-9721-9c8898af37e8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkcnw\" (UniqueName: \"kubernetes.io/projected/5ad10c96-2142-41c0-836a-7622bf8767bb-kube-api-access-rkcnw\") pod \"iptables-alerter-q6wmj\" (UID: \"5ad10c96-2142-41c0-836a-7622bf8767bb\") " pod="openshift-network-operator/iptables-alerter-q6wmj"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5ad10c96-2142-41c0-836a-7622bf8767bb-iptables-alerter-script\") pod \"iptables-alerter-q6wmj\" (UID: \"5ad10c96-2142-41c0-836a-7622bf8767bb\") " pod="openshift-network-operator/iptables-alerter-q6wmj"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-env-overrides\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-host-cni-bin\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-var-lib-kubelet\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.565857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-sysctl-d\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:28.565901    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:28.565979    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs podName:a04476a3-6d83-40ff-879c-c9a5b620ef44 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:29.065941903 +0000 UTC m=+2.990853391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs") pod "network-metrics-daemon-vj4n4" (UID: "a04476a3-6d83-40ff-879c-c9a5b620ef44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.566083    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-systemd\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.566125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-run-k8s-cni-cncf-io\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.567711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.566163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-var-lib-cni-multus\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.568546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.566187    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-cnibin\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.568546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.566200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-host-run-multus-certs\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.568546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.566205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-var-lib-openvswitch\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.568546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.566219    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77f37552-1d26-4529-bdb1-88d5f2b3af43-multus-socket-dir-parent\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.568546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.566236    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/688eff1f-bb3f-4223-9721-9c8898af37e8-cnibin\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.568546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.566240    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-etc-openvswitch\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.568546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.566271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-host\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.568546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.567782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/851a86dd-ec09-426e-a81e-71c143dacb4f-agent-certs\") pod \"konnectivity-agent-sc5dh\" (UID: \"851a86dd-ec09-426e-a81e-71c143dacb4f\") " pod="kube-system/konnectivity-agent-sc5dh"
Sep 10 12:30:28.568546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.568438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-tmp\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.568546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.568544    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-ovn-node-metrics-cert\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.571147 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.571117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-etc-tuned\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.573610 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:28.573583    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:30:28.573610 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:28.573608    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:30:28.573769 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:28.573622    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5rhmq for pod openshift-network-diagnostics/network-check-target-jz9tr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:28.573769 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:28.573685    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq podName:911473b1-59c6-4916-a2d6-2dca448752a2 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:29.073667512 +0000 UTC m=+2.998579001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5rhmq" (UniqueName: "kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq") pod "network-check-target-jz9tr" (UID: "911473b1-59c6-4916-a2d6-2dca448752a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:28.573936 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.573914    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgk9k\" (UniqueName: \"kubernetes.io/projected/5d9368c9-0d47-4aa9-b074-4977e5d2e66b-kube-api-access-tgk9k\") pod \"tuned-q9g7f\" (UID: \"5d9368c9-0d47-4aa9-b074-4977e5d2e66b\") " pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.575053 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.575003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkcnw\" (UniqueName: \"kubernetes.io/projected/5ad10c96-2142-41c0-836a-7622bf8767bb-kube-api-access-rkcnw\") pod \"iptables-alerter-q6wmj\" (UID: \"5ad10c96-2142-41c0-836a-7622bf8767bb\") " pod="openshift-network-operator/iptables-alerter-q6wmj"
Sep 10 12:30:28.575293 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.575270    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6kwz\" (UniqueName: \"kubernetes.io/projected/77f37552-1d26-4529-bdb1-88d5f2b3af43-kube-api-access-m6kwz\") pod \"multus-484p6\" (UID: \"77f37552-1d26-4529-bdb1-88d5f2b3af43\") " pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.576068 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.576045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wnpl\" (UniqueName: \"kubernetes.io/projected/688eff1f-bb3f-4223-9721-9c8898af37e8-kube-api-access-2wnpl\") pod \"multus-additional-cni-plugins-ntbv6\" (UID: \"688eff1f-bb3f-4223-9721-9c8898af37e8\") " pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.576242 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.576220    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtpvz\" (UniqueName: \"kubernetes.io/projected/8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5-kube-api-access-dtpvz\") pod \"ovnkube-node-dtnsr\" (UID: \"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5\") " pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.576887 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.576828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lkk9\" (UniqueName: \"kubernetes.io/projected/c1e745f6-42ce-443e-8555-1572dbad7050-kube-api-access-2lkk9\") pod \"node-ca-n5899\" (UID: \"c1e745f6-42ce-443e-8555-1572dbad7050\") " pod="openshift-image-registry/node-ca-n5899"
Sep 10 12:30:28.577167 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.577148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2t2s\" (UniqueName: \"kubernetes.io/projected/a04476a3-6d83-40ff-879c-c9a5b620ef44-kube-api-access-m2t2s\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:28.634943 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.634909    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:30:28.666226 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666186    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-socket-dir\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.666226 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-etc-selinux\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.666445 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4be22799-8b7d-4e18-867f-34a3185af15a-dbus\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:28.666445 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666400    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-etc-selinux\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.666536 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.666536 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4be22799-8b7d-4e18-867f-34a3185af15a-dbus\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:28.666536 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-socket-dir\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.666696 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-sys-fs\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.666696 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.666696 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcnn7\" (UniqueName: \"kubernetes.io/projected/a6e7308e-da70-4a2b-845a-51057e2f1f94-kube-api-access-tcnn7\") pod \"node-resolver-j7fz9\" (UID: \"a6e7308e-da70-4a2b-845a-51057e2f1f94\") " pod="openshift-dns/node-resolver-j7fz9"
Sep 10 12:30:28.666696 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-registration-dir\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.666868 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-sys-fs\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.666868 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6e7308e-da70-4a2b-845a-51057e2f1f94-hosts-file\") pod \"node-resolver-j7fz9\" (UID: \"a6e7308e-da70-4a2b-845a-51057e2f1f94\") " pod="openshift-dns/node-resolver-j7fz9"
Sep 10 12:30:28.666868 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666835    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-registration-dir\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.666868 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lnjw\" (UniqueName: \"kubernetes.io/projected/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-kube-api-access-2lnjw\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.667049 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6e7308e-da70-4a2b-845a-51057e2f1f94-tmp-dir\") pod \"node-resolver-j7fz9\" (UID: \"a6e7308e-da70-4a2b-845a-51057e2f1f94\") " pod="openshift-dns/node-resolver-j7fz9"
Sep 10 12:30:28.667049 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666888    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6e7308e-da70-4a2b-845a-51057e2f1f94-hosts-file\") pod \"node-resolver-j7fz9\" (UID: \"a6e7308e-da70-4a2b-845a-51057e2f1f94\") " pod="openshift-dns/node-resolver-j7fz9"
Sep 10 12:30:28.667049 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-device-dir\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.667049 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.666996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4be22799-8b7d-4e18-867f-34a3185af15a-kubelet-config\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:28.667049 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.667022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:28.667282 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.667082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-device-dir\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.667282 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.667100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4be22799-8b7d-4e18-867f-34a3185af15a-kubelet-config\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:28.667282 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:28.667179    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:28.667282 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.667223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6e7308e-da70-4a2b-845a-51057e2f1f94-tmp-dir\") pod \"node-resolver-j7fz9\" (UID: \"a6e7308e-da70-4a2b-845a-51057e2f1f94\") " pod="openshift-dns/node-resolver-j7fz9"
Sep 10 12:30:28.667282 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:28.667243    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret podName:4be22799-8b7d-4e18-867f-34a3185af15a nodeName:}" failed. No retries permitted until 2026-09-10 12:30:29.167223143 +0000 UTC m=+3.092134610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret") pod "global-pull-secret-syncer-jfpm9" (UID: "4be22799-8b7d-4e18-867f-34a3185af15a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:28.676276 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.676204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcnn7\" (UniqueName: \"kubernetes.io/projected/a6e7308e-da70-4a2b-845a-51057e2f1f94-kube-api-access-tcnn7\") pod \"node-resolver-j7fz9\" (UID: \"a6e7308e-da70-4a2b-845a-51057e2f1f94\") " pod="openshift-dns/node-resolver-j7fz9"
Sep 10 12:30:28.676470 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.676449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lnjw\" (UniqueName: \"kubernetes.io/projected/ac0e6a55-ef6f-4f2f-98d4-12992e1e7148-kube-api-access-2lnjw\") pod \"aws-ebs-csi-driver-node-zqssh\" (UID: \"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.702495 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.702439    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-150.ec2.internal" event={"ID":"87b3b84e20b1589782e644bb9f5ff7c5","Type":"ContainerStarted","Data":"e97f7a1f696159aef7fe9eaa39d963ae73935a98cfeab0d331d2eae3921396ba"}
Sep 10 12:30:28.703720 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.703689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal" event={"ID":"f3a4ab1e3c1464e6c152bc80398459b4","Type":"ContainerStarted","Data":"51cf56d598f88b38da83a8b5ceedd0fa12cf22b5c160489a99356aed8ce2ede4"}
Sep 10 12:30:28.750463 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.750434    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:28.761859 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.761831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q9g7f"
Sep 10 12:30:28.771054 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.771031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n5899"
Sep 10 12:30:28.778089 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.778069    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-484p6"
Sep 10 12:30:28.786040 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.786018    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ntbv6"
Sep 10 12:30:28.794989 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.794969    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sc5dh"
Sep 10 12:30:28.803051 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.803031    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q6wmj"
Sep 10 12:30:28.811978 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.811958    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh"
Sep 10 12:30:28.818724 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:28.818702    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j7fz9"
Sep 10 12:30:29.070088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.070004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:29.070217 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:29.070137    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:29.070217 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:29.070193    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs podName:a04476a3-6d83-40ff-879c-c9a5b620ef44 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:30.070179431 +0000 UTC m=+3.995090911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs") pod "network-metrics-daemon-vj4n4" (UID: "a04476a3-6d83-40ff-879c-c9a5b620ef44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:29.164564 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.164519    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:30:29.171064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.171031    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhmq\" (UniqueName: \"kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq\") pod \"network-check-target-jz9tr\" (UID: \"911473b1-59c6-4916-a2d6-2dca448752a2\") " pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:29.171220 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.171121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:29.171296 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:29.171219    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:30:29.171296 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:29.171245    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:30:29.171296 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:29.171257    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5rhmq for pod openshift-network-diagnostics/network-check-target-jz9tr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:29.171416 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:29.171219    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:29.171416 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:29.171317    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq podName:911473b1-59c6-4916-a2d6-2dca448752a2 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:30.171302414 +0000 UTC m=+4.096213879 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5rhmq" (UniqueName: "kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq") pod "network-check-target-jz9tr" (UID: "911473b1-59c6-4916-a2d6-2dca448752a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:29.171416 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:29.171389    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret podName:4be22799-8b7d-4e18-867f-34a3185af15a nodeName:}" failed. No retries permitted until 2026-09-10 12:30:30.171373328 +0000 UTC m=+4.096284796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret") pod "global-pull-secret-syncer-jfpm9" (UID: "4be22799-8b7d-4e18-867f-34a3185af15a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:29.490518 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.490424    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 12:25:28 +0000 UTC" deadline="2028-05-19 19:28:45.587633258 +0000 UTC"
Sep 10 12:30:29.490518 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.490456    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14814h58m16.097181268s"
Sep 10 12:30:29.683784 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:29.683569    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac0e6a55_ef6f_4f2f_98d4_12992e1e7148.slice/crio-91d4c7e26cbaee586c020d12de4c67b22461cfb19f88e4080a4c99f02e350576 WatchSource:0}: Error finding container 91d4c7e26cbaee586c020d12de4c67b22461cfb19f88e4080a4c99f02e350576: Status 404 returned error can't find the container with id 91d4c7e26cbaee586c020d12de4c67b22461cfb19f88e4080a4c99f02e350576
Sep 10 12:30:29.684602 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:29.684578    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e745f6_42ce_443e_8555_1572dbad7050.slice/crio-6572c9bf408c72fd3e31a6b1a0040e6680e7ee06bdc7d0b5b7e1b6359ea40b43 WatchSource:0}: Error finding container 6572c9bf408c72fd3e31a6b1a0040e6680e7ee06bdc7d0b5b7e1b6359ea40b43: Status 404 returned error can't find the container with id 6572c9bf408c72fd3e31a6b1a0040e6680e7ee06bdc7d0b5b7e1b6359ea40b43
Sep 10 12:30:29.685413 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:29.685213    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad10c96_2142_41c0_836a_7622bf8767bb.slice/crio-d504c614f1cdc801feb5dac3e201ac6672840c17512ee3f552c4e68bd7f5a71b WatchSource:0}: Error finding container d504c614f1cdc801feb5dac3e201ac6672840c17512ee3f552c4e68bd7f5a71b: Status 404 returned error can't find the container with id d504c614f1cdc801feb5dac3e201ac6672840c17512ee3f552c4e68bd7f5a71b
Sep 10 12:30:29.687577 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:29.687533    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688eff1f_bb3f_4223_9721_9c8898af37e8.slice/crio-5f2527cc61a3b1f8f3a0f27b05b5c47905edbce1488fc2009a76ba7558390e23 WatchSource:0}: Error finding container 5f2527cc61a3b1f8f3a0f27b05b5c47905edbce1488fc2009a76ba7558390e23: Status 404 returned error can't find the container with id 5f2527cc61a3b1f8f3a0f27b05b5c47905edbce1488fc2009a76ba7558390e23
Sep 10 12:30:29.688801 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:30:29.688682    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851a86dd_ec09_426e_a81e_71c143dacb4f.slice/crio-5a0eea7a93d8744118e41b954cb7b0a951cef1d8b46fb319e25adf8c40036252 WatchSource:0}: Error finding container 5a0eea7a93d8744118e41b954cb7b0a951cef1d8b46fb319e25adf8c40036252: Status 404 returned error can't find the container with id 5a0eea7a93d8744118e41b954cb7b0a951cef1d8b46fb319e25adf8c40036252
Sep 10 12:30:29.698394 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.698115    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:29.698394 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:29.698253    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:29.706656 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.706625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntbv6" event={"ID":"688eff1f-bb3f-4223-9721-9c8898af37e8","Type":"ContainerStarted","Data":"5f2527cc61a3b1f8f3a0f27b05b5c47905edbce1488fc2009a76ba7558390e23"}
Sep 10 12:30:29.708243 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.708202    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q6wmj" event={"ID":"5ad10c96-2142-41c0-836a-7622bf8767bb","Type":"ContainerStarted","Data":"d504c614f1cdc801feb5dac3e201ac6672840c17512ee3f552c4e68bd7f5a71b"}
Sep 10 12:30:29.709643 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.709611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n5899" event={"ID":"c1e745f6-42ce-443e-8555-1572dbad7050","Type":"ContainerStarted","Data":"6572c9bf408c72fd3e31a6b1a0040e6680e7ee06bdc7d0b5b7e1b6359ea40b43"}
Sep 10 12:30:29.713542 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.713513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh" event={"ID":"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148","Type":"ContainerStarted","Data":"91d4c7e26cbaee586c020d12de4c67b22461cfb19f88e4080a4c99f02e350576"}
Sep 10 12:30:29.715021 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.714917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7fz9" event={"ID":"a6e7308e-da70-4a2b-845a-51057e2f1f94","Type":"ContainerStarted","Data":"feeed6e1e2db1e0db15bb984d688a556e39acb438003269f4cec40f82713b8eb"}
Sep 10 12:30:29.716484 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.716440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr" event={"ID":"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5","Type":"ContainerStarted","Data":"550d58ef7800cab37b4472afe7099a6070e694a4b3d46b113fb7f77b3db4a771"}
Sep 10 12:30:29.718167 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.718140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sc5dh" event={"ID":"851a86dd-ec09-426e-a81e-71c143dacb4f","Type":"ContainerStarted","Data":"5a0eea7a93d8744118e41b954cb7b0a951cef1d8b46fb319e25adf8c40036252"}
Sep 10 12:30:29.719448 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.719426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-484p6" event={"ID":"77f37552-1d26-4529-bdb1-88d5f2b3af43","Type":"ContainerStarted","Data":"e0ac08072809927178730c96f108ad5f6109a0f2f7ba24ee0737a28720471939"}
Sep 10 12:30:29.721279 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:29.721044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q9g7f" event={"ID":"5d9368c9-0d47-4aa9-b074-4977e5d2e66b","Type":"ContainerStarted","Data":"f7ab23eb1945aeaf20220dd903cdf151f7ce79a82c5a06bdb55d2001202c21ac"}
Sep 10 12:30:30.078255 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:30.078163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:30.078412 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:30.078339    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:30.078412 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:30.078402    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs podName:a04476a3-6d83-40ff-879c-c9a5b620ef44 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:32.078382712 +0000 UTC m=+6.003294181 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs") pod "network-metrics-daemon-vj4n4" (UID: "a04476a3-6d83-40ff-879c-c9a5b620ef44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:30.178634 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:30.178573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:30.178634 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:30.178637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhmq\" (UniqueName: \"kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq\") pod \"network-check-target-jz9tr\" (UID: \"911473b1-59c6-4916-a2d6-2dca448752a2\") " pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:30.178872 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:30.178804    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:30:30.178872 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:30.178823    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:30:30.178872 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:30.178836    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5rhmq for pod openshift-network-diagnostics/network-check-target-jz9tr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:30.179028 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:30.178893    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq podName:911473b1-59c6-4916-a2d6-2dca448752a2 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:32.178874902 +0000 UTC m=+6.103786373 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5rhmq" (UniqueName: "kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq") pod "network-check-target-jz9tr" (UID: "911473b1-59c6-4916-a2d6-2dca448752a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:30.179287 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:30.179268    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:30.179362 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:30.179325    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret podName:4be22799-8b7d-4e18-867f-34a3185af15a nodeName:}" failed. No retries permitted until 2026-09-10 12:30:32.179307618 +0000 UTC m=+6.104219098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret") pod "global-pull-secret-syncer-jfpm9" (UID: "4be22799-8b7d-4e18-867f-34a3185af15a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:30.490719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:30.490658    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 12:25:28 +0000 UTC" deadline="2028-05-18 04:33:48.349166212 +0000 UTC"
Sep 10 12:30:30.490719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:30.490716    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14776h3m17.858455033s"
Sep 10 12:30:30.699700 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:30.699668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:30.699883 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:30.699797    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:30.700673 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:30.700647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:30.700788 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:30.700750    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:30.750204 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:30.750100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-150.ec2.internal" event={"ID":"87b3b84e20b1589782e644bb9f5ff7c5","Type":"ContainerStarted","Data":"14a62bb486ed1bbea0499134e06660c181fa1ba5f6bc8e8a644e92757f91fb62"}
Sep 10 12:30:30.755516 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:30.755482    2563 generic.go:356] "Generic (PLEG): container finished" podID="f3a4ab1e3c1464e6c152bc80398459b4" containerID="dd8374dcf611322b1d2f81993bc2ef6aaa3764530d81cf88d519bbc8413d7ea2" exitCode=0
Sep 10 12:30:30.755682 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:30.755540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal" event={"ID":"f3a4ab1e3c1464e6c152bc80398459b4","Type":"ContainerDied","Data":"dd8374dcf611322b1d2f81993bc2ef6aaa3764530d81cf88d519bbc8413d7ea2"}
Sep 10 12:30:30.783870 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:30.783780    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-150.ec2.internal" podStartSLOduration=3.783758914 podStartE2EDuration="3.783758914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:30.76654626 +0000 UTC m=+4.691457750" watchObservedRunningTime="2026-09-10 12:30:30.783758914 +0000 UTC m=+4.708670406"
Sep 10 12:30:31.701151 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:31.698753    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:31.701151 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:31.698895    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:31.781517 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:31.781438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal" event={"ID":"f3a4ab1e3c1464e6c152bc80398459b4","Type":"ContainerStarted","Data":"ad74f765b5ecf8fa0d478c5c9b94f7bd0bb0356b1c13b5057d0adb5d9fd5061c"}
Sep 10 12:30:32.098498 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:32.098416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:32.098712 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:32.098633    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:32.098712 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:32.098694    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs podName:a04476a3-6d83-40ff-879c-c9a5b620ef44 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:36.098675067 +0000 UTC m=+10.023586538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs") pod "network-metrics-daemon-vj4n4" (UID: "a04476a3-6d83-40ff-879c-c9a5b620ef44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:32.200383 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:32.199522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhmq\" (UniqueName: \"kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq\") pod \"network-check-target-jz9tr\" (UID: \"911473b1-59c6-4916-a2d6-2dca448752a2\") " pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:32.200383 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:32.199632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:32.200383 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:32.199817    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:32.200383 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:32.199887    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret podName:4be22799-8b7d-4e18-867f-34a3185af15a nodeName:}" failed. No retries permitted until 2026-09-10 12:30:36.199868244 +0000 UTC m=+10.124779712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret") pod "global-pull-secret-syncer-jfpm9" (UID: "4be22799-8b7d-4e18-867f-34a3185af15a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:32.200383 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:32.199981    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:30:32.200383 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:32.199999    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:30:32.200383 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:32.200011    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5rhmq for pod openshift-network-diagnostics/network-check-target-jz9tr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:32.200383 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:32.200048    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq podName:911473b1-59c6-4916-a2d6-2dca448752a2 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:36.200036186 +0000 UTC m=+10.124947657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5rhmq" (UniqueName: "kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq") pod "network-check-target-jz9tr" (UID: "911473b1-59c6-4916-a2d6-2dca448752a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:32.700767 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:32.699450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:32.700767 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:32.699611    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:32.700767 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:32.699672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:32.700767 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:32.699799    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:33.698640 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:33.698544    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:33.699123 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:33.698759    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:34.699433 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:34.698803    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:34.699433 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:34.698930    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:34.699433 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:34.699307    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:34.699433 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:34.699391    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:35.698092 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:35.697953    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:35.698092 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:35.698091    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:36.135460 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:36.135376    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:36.135881 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:36.135569    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:36.135881 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:36.135644    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs podName:a04476a3-6d83-40ff-879c-c9a5b620ef44 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:44.135623473 +0000 UTC m=+18.060534953 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs") pod "network-metrics-daemon-vj4n4" (UID: "a04476a3-6d83-40ff-879c-c9a5b620ef44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:36.236541 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:36.236144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhmq\" (UniqueName: \"kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq\") pod \"network-check-target-jz9tr\" (UID: \"911473b1-59c6-4916-a2d6-2dca448752a2\") " pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:36.236541 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:36.236250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:36.236541 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:36.236386    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:36.236541 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:36.236448    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:30:36.236541 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:36.236473    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:30:36.236541 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:36.236487    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5rhmq for pod openshift-network-diagnostics/network-check-target-jz9tr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:36.236541 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:36.236455    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret podName:4be22799-8b7d-4e18-867f-34a3185af15a nodeName:}" failed. No retries permitted until 2026-09-10 12:30:44.236433565 +0000 UTC m=+18.161345054 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret") pod "global-pull-secret-syncer-jfpm9" (UID: "4be22799-8b7d-4e18-867f-34a3185af15a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:36.236541 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:36.236564    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq podName:911473b1-59c6-4916-a2d6-2dca448752a2 nodeName:}" failed. No retries permitted until 2026-09-10 12:30:44.236534237 +0000 UTC m=+18.161445707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5rhmq" (UniqueName: "kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq") pod "network-check-target-jz9tr" (UID: "911473b1-59c6-4916-a2d6-2dca448752a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:36.700517 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:36.699919    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:36.700517 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:36.700042    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:36.700517 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:36.700379    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:36.700517 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:36.700473    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:37.698311 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:37.698153    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:37.698311 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:37.698298    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:38.698716 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:38.698495    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:38.699164 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:38.698541    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:38.699164 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:38.698819    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:38.699164 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:38.698864    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:39.698595 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:39.698566    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:39.698779 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:39.698671    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:40.698431 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:40.698393    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:40.698625 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:40.698532    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:40.698625 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:40.698587    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:40.698727 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:40.698701    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:41.698388 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:41.698347    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:41.698858 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:41.698490    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:42.698006 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:42.697970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:42.698194 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:42.698130    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:42.698194 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:42.698163    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:42.698303 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:42.698266    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:43.698561 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:43.698512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:43.699053 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:43.698652    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:44.198972 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:44.198932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:44.199145 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:44.199075    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:44.199216 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:44.199156    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs podName:a04476a3-6d83-40ff-879c-c9a5b620ef44 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:00.199131427 +0000 UTC m=+34.124042915 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs") pod "network-metrics-daemon-vj4n4" (UID: "a04476a3-6d83-40ff-879c-c9a5b620ef44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:30:44.299822 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:44.299785    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:44.300006 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:44.299840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhmq\" (UniqueName: \"kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq\") pod \"network-check-target-jz9tr\" (UID: \"911473b1-59c6-4916-a2d6-2dca448752a2\") " pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:44.300006 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:44.299952    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:44.300006 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:44.299990    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:30:44.300145 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:44.300009    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:30:44.300145 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:44.300022    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5rhmq for pod openshift-network-diagnostics/network-check-target-jz9tr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:44.300145 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:44.300027    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret podName:4be22799-8b7d-4e18-867f-34a3185af15a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:00.300009477 +0000 UTC m=+34.224920946 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret") pod "global-pull-secret-syncer-jfpm9" (UID: "4be22799-8b7d-4e18-867f-34a3185af15a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:30:44.300145 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:44.300072    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq podName:911473b1-59c6-4916-a2d6-2dca448752a2 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:00.300057016 +0000 UTC m=+34.224968497 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5rhmq" (UniqueName: "kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq") pod "network-check-target-jz9tr" (UID: "911473b1-59c6-4916-a2d6-2dca448752a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:30:44.698092 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:44.698054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:44.698274 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:44.698054    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:44.698274 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:44.698198    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:44.698395 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:44.698270    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:45.698535 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:45.698501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:45.699006 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:45.698653    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:46.699569 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:46.699526    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:46.700116 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:46.699641    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:46.700116 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:46.699661    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:46.700116 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:46.699768    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:47.698273 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.698094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:47.698385 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:47.698367    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:47.816030 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.815996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n5899" event={"ID":"c1e745f6-42ce-443e-8555-1572dbad7050","Type":"ContainerStarted","Data":"3378e75e6076ea6324286474a686ff8a8008514b08b1fac001a7d1463ef735d1"}
Sep 10 12:30:47.817665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.817643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh" event={"ID":"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148","Type":"ContainerStarted","Data":"0001782f6a9a8cfe77d6772515843df004316ba81fd7411f4445b14b81044149"}
Sep 10 12:30:47.818930 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.818910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j7fz9" event={"ID":"a6e7308e-da70-4a2b-845a-51057e2f1f94","Type":"ContainerStarted","Data":"b9bc807d61034a62e037ec0d181f7f99d7d5380600f31c4bb9720405206e1e49"}
Sep 10 12:30:47.822142 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.822122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr" event={"ID":"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5","Type":"ContainerStarted","Data":"3b514a4dfdcec9ded62d2f33c1122e9086907d60780e61787b6533458e341053"}
Sep 10 12:30:47.822244 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.822192    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr" event={"ID":"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5","Type":"ContainerStarted","Data":"5cac197c008d15207c60365f3ab200149c9ece4b03b9bfc8ca437d899a457303"}
Sep 10 12:30:47.822301 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.822246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr" event={"ID":"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5","Type":"ContainerStarted","Data":"140dec682f5f3e032fc8089d8b9ce949a5787f54acd8363683fca17e87bc3a59"}
Sep 10 12:30:47.822301 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.822295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr" event={"ID":"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5","Type":"ContainerStarted","Data":"a0e5555067244ce9d298edf89dc530c2f033f896c6f94d8c4382d73810f112bd"}
Sep 10 12:30:47.822402 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.822342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr" event={"ID":"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5","Type":"ContainerStarted","Data":"44cc6be78d901db363eafb52ff84e084cfb96359da3b909f76b139a79f8d80f1"}
Sep 10 12:30:47.822402 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.822389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr" event={"ID":"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5","Type":"ContainerStarted","Data":"470db4934e3cdb01a5eb4b9e45f58849f5e2f7f4b0a3733b5cc2efa28f224300"}
Sep 10 12:30:47.823506 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.823487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sc5dh" event={"ID":"851a86dd-ec09-426e-a81e-71c143dacb4f","Type":"ContainerStarted","Data":"ce6599546860f3a80dc2277b2ba19faa5f485c21260ebeaeee352be29281d8ba"}
Sep 10 12:30:47.824995 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.824971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-484p6" event={"ID":"77f37552-1d26-4529-bdb1-88d5f2b3af43","Type":"ContainerStarted","Data":"fe7347e284cc36539e395b05dc18e2ef14559c84c290678177f80dcc735bee35"}
Sep 10 12:30:47.826585 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.826532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q9g7f" event={"ID":"5d9368c9-0d47-4aa9-b074-4977e5d2e66b","Type":"ContainerStarted","Data":"b5afa18598d6e7ebd9be5711f4958e8eda22a1ab40002019ef1b0eae1a681b4d"}
Sep 10 12:30:47.828285 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.828258    2563 generic.go:356] "Generic (PLEG): container finished" podID="688eff1f-bb3f-4223-9721-9c8898af37e8" containerID="22ce45f49c60340383fa0122a146fb846106a99837496977fa669c9f6fafbd5e" exitCode=0
Sep 10 12:30:47.828379 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.828294    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntbv6" event={"ID":"688eff1f-bb3f-4223-9721-9c8898af37e8","Type":"ContainerDied","Data":"22ce45f49c60340383fa0122a146fb846106a99837496977fa669c9f6fafbd5e"}
Sep 10 12:30:47.832042 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.832001    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-150.ec2.internal" podStartSLOduration=20.83198524 podStartE2EDuration="20.83198524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:31.79946411 +0000 UTC m=+5.724375600" watchObservedRunningTime="2026-09-10 12:30:47.83198524 +0000 UTC m=+21.756896730"
Sep 10 12:30:47.832308 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.832279    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n5899" podStartSLOduration=4.306963387 podStartE2EDuration="21.832273343s" totalImagesPullingTime="17.525309956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:47.832114506 +0000 UTC m=+21.757025984" watchObservedRunningTime="2026-09-10 12:30:47.832273343 +0000 UTC m=+21.757184831"
Sep 10 12:30:47.849872 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.849828    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-484p6" podStartSLOduration=4.119840726 podStartE2EDuration="21.849814323s" totalImagesPullingTime="17.729973597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:47.849270348 +0000 UTC m=+21.774181838" watchObservedRunningTime="2026-09-10 12:30:47.849814323 +0000 UTC m=+21.774725812"
Sep 10 12:30:47.865626 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.865570    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sc5dh" podStartSLOduration=4.439918539 podStartE2EDuration="21.865538163s" totalImagesPullingTime="17.425619624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:47.865048044 +0000 UTC m=+21.789959532" watchObservedRunningTime="2026-09-10 12:30:47.865538163 +0000 UTC m=+21.790449642"
Sep 10 12:30:47.881507 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.881429    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q9g7f" podStartSLOduration=4.363836609 podStartE2EDuration="21.881416002s" totalImagesPullingTime="17.517579393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:47.881358625 +0000 UTC m=+21.806270113" watchObservedRunningTime="2026-09-10 12:30:47.881416002 +0000 UTC m=+21.806327490"
Sep 10 12:30:47.920410 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:47.920356    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j7fz9" podStartSLOduration=4.401587549 podStartE2EDuration="21.920339873s" totalImagesPullingTime="17.518752324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:47.919907528 +0000 UTC m=+21.844819015" watchObservedRunningTime="2026-09-10 12:30:47.920339873 +0000 UTC m=+21.845251361"
Sep 10 12:30:48.022935 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:48.022903    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sc5dh"
Sep 10 12:30:48.023609 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:48.023578    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sc5dh"
Sep 10 12:30:48.698822 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:48.698787    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:48.699036 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:48.698789    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:48.699036 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:48.698919    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:48.699036 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:48.699004    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:48.833586 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:48.832838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q6wmj" event={"ID":"5ad10c96-2142-41c0-836a-7622bf8767bb","Type":"ContainerStarted","Data":"2675c456f0d09942f18dd7b3576d983fce19cffe186c7962b574b88586cd2360"}
Sep 10 12:30:48.834015 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:48.833737    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sc5dh"
Sep 10 12:30:48.834318 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:48.834291    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sc5dh"
Sep 10 12:30:48.871327 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:48.871274    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q6wmj" podStartSLOduration=5.345124865 podStartE2EDuration="22.871253188s" totalImagesPullingTime="17.526128323s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:48.848845013 +0000 UTC m=+22.773756500" watchObservedRunningTime="2026-09-10 12:30:48.871253188 +0000 UTC m=+22.796164677"
Sep 10 12:30:48.897913 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:48.897887    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 12:30:49.509519 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:49.509424    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T12:30:48.897909232Z","UUID":"732bb00a-fcb8-4509-91ad-c86df4f56e35","Handler":null,"Name":"","Endpoint":""}
Sep 10 12:30:49.511218 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:49.511194    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 12:30:49.511348 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:49.511228    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 12:30:49.698660 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:49.698623    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:49.698822 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:49.698756    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:49.837577 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:49.837475    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh" event={"ID":"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148","Type":"ContainerStarted","Data":"ae88fd57a22d7a4bea1c05afcdafdd05d11771d836c84bd6561c5078783f196a"}
Sep 10 12:30:50.698109 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:50.698077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:50.698284 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:50.698204    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:50.698506 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:50.698077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:50.698506 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:50.698474    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:50.843893 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:50.843802    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr" event={"ID":"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5","Type":"ContainerStarted","Data":"438dc0e5fca05c8088c9043c17c7526dc8eb9c3c6fefebfcb4f8f1e93179b36e"}
Sep 10 12:30:50.847745 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:50.847711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh" event={"ID":"ac0e6a55-ef6f-4f2f-98d4-12992e1e7148","Type":"ContainerStarted","Data":"015562908fe623d097081a72fb867d6d2292678f6bad335c5817b0d2455d58ef"}
Sep 10 12:30:50.866041 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:50.865985    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zqssh" podStartSLOduration=4.415048776 podStartE2EDuration="24.865968184s" totalImagesPullingTime="20.450919408s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:50.865513086 +0000 UTC m=+24.790424573" watchObservedRunningTime="2026-09-10 12:30:50.865968184 +0000 UTC m=+24.790879672"
Sep 10 12:30:51.698467 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:51.698435    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:51.698682 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:51.698575    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:52.699379 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:52.699211    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:52.699788 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:52.699227    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:52.699788 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:52.699482    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:52.699788 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:52.699625    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:52.854893 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:52.854854    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntbv6" event={"ID":"688eff1f-bb3f-4223-9721-9c8898af37e8","Type":"ContainerStarted","Data":"0d198a2f7aa370fa1ae7dc0c1ec818a977b175f2110639f31e1841efbffbf68d"}
Sep 10 12:30:52.858630 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:52.858603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr" event={"ID":"8a0a6e67-8eed-4fbc-b4d6-e7d1a1ddcac5","Type":"ContainerStarted","Data":"e5749fd4b50a03af01dd8f6fcaa50e188527f2e0d36e497865109bce3bf85fa5"}
Sep 10 12:30:52.858936 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:52.858918    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:52.873462 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:52.873433    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:52.899794 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:52.899749    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr" podStartSLOduration=9.377980375 podStartE2EDuration="26.899732571s" totalImagesPullingTime="17.521752196s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:52.899531734 +0000 UTC m=+26.824443225" watchObservedRunningTime="2026-09-10 12:30:52.899732571 +0000 UTC m=+26.824644059"
Sep 10 12:30:53.698487 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:53.698446    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:53.698732 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:53.698634    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:53.863888 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:53.863848    2563 generic.go:356] "Generic (PLEG): container finished" podID="688eff1f-bb3f-4223-9721-9c8898af37e8" containerID="0d198a2f7aa370fa1ae7dc0c1ec818a977b175f2110639f31e1841efbffbf68d" exitCode=0
Sep 10 12:30:53.864382 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:53.863893    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntbv6" event={"ID":"688eff1f-bb3f-4223-9721-9c8898af37e8","Type":"ContainerDied","Data":"0d198a2f7aa370fa1ae7dc0c1ec818a977b175f2110639f31e1841efbffbf68d"}
Sep 10 12:30:53.864663 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:53.864518    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:53.864663 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:53.864611    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:53.880331 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:53.880305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:30:54.471401 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:54.471323    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vj4n4"]
Sep 10 12:30:54.472196 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:54.471783    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:54.472196 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:54.471911    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:54.472196 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:54.472029    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jz9tr"]
Sep 10 12:30:54.472196 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:54.472107    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:54.472196 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:54.472177    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:54.485451 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:54.485384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jfpm9"]
Sep 10 12:30:54.485611 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:54.485497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:54.485678 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:54.485614    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:54.868296 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:54.868104    2563 generic.go:356] "Generic (PLEG): container finished" podID="688eff1f-bb3f-4223-9721-9c8898af37e8" containerID="53db370b215e53e1910d0b6c9f658b1ac689ad57343af2de3da41d5c49bf27d9" exitCode=0
Sep 10 12:30:54.868904 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:54.868183    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntbv6" event={"ID":"688eff1f-bb3f-4223-9721-9c8898af37e8","Type":"ContainerDied","Data":"53db370b215e53e1910d0b6c9f658b1ac689ad57343af2de3da41d5c49bf27d9"}
Sep 10 12:30:55.698540 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:55.698324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:55.698540 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:55.698497    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:55.873247 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:55.873156    2563 generic.go:356] "Generic (PLEG): container finished" podID="688eff1f-bb3f-4223-9721-9c8898af37e8" containerID="6895c5f600095c159fa3ccc8774c9df16ff0d575c4ee6d518b62ff464dd761d9" exitCode=0
Sep 10 12:30:55.873247 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:55.873234    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntbv6" event={"ID":"688eff1f-bb3f-4223-9721-9c8898af37e8","Type":"ContainerDied","Data":"6895c5f600095c159fa3ccc8774c9df16ff0d575c4ee6d518b62ff464dd761d9"}
Sep 10 12:30:56.699098 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:56.699065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:56.699275 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:56.699178    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:56.699392 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:56.699367    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:56.699580 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:56.699500    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:57.698645 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:57.698613    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:57.699049 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:57.698713    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:30:58.698182 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:58.698149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:30:58.698345 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:58.698299    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vj4n4" podUID="a04476a3-6d83-40ff-879c-c9a5b620ef44"
Sep 10 12:30:58.698345 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:58.698336    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:30:58.700969 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:58.698726    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jfpm9" podUID="4be22799-8b7d-4e18-867f-34a3185af15a"
Sep 10 12:30:59.698411 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:30:59.698379    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:30:59.698611 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:30:59.698495    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jz9tr" podUID="911473b1-59c6-4916-a2d6-2dca448752a2"
Sep 10 12:31:00.216054 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.216022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:31:00.216409 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:00.216149    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:31:00.216409 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:00.216201    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs podName:a04476a3-6d83-40ff-879c-c9a5b620ef44 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:32.216187551 +0000 UTC m=+66.141099030 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs") pod "network-metrics-daemon-vj4n4" (UID: "a04476a3-6d83-40ff-879c-c9a5b620ef44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:31:00.305004 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.304977    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-150.ec2.internal" event="NodeReady"
Sep 10 12:31:00.305136 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.305110    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 12:31:00.316570 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.316525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:31:00.316699 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.316591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhmq\" (UniqueName: \"kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq\") pod \"network-check-target-jz9tr\" (UID: \"911473b1-59c6-4916-a2d6-2dca448752a2\") " pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:31:00.316699 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:00.316677    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:31:00.316823 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:00.316732    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret podName:4be22799-8b7d-4e18-867f-34a3185af15a nodeName:}" failed. No retries permitted until 2026-09-10 12:31:32.316718392 +0000 UTC m=+66.241629884 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret") pod "global-pull-secret-syncer-jfpm9" (UID: "4be22799-8b7d-4e18-867f-34a3185af15a") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:31:00.316823 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:00.316735    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:31:00.316823 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:00.316753    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:31:00.316823 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:00.316767    2563 projected.go:196] Error preparing data for projected volume kube-api-access-5rhmq for pod openshift-network-diagnostics/network-check-target-jz9tr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:31:00.316823 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:00.316814    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq podName:911473b1-59c6-4916-a2d6-2dca448752a2 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:32.316799133 +0000 UTC m=+66.241710612 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5rhmq" (UniqueName: "kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq") pod "network-check-target-jz9tr" (UID: "911473b1-59c6-4916-a2d6-2dca448752a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:31:00.351452 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.351422    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5qwhw"]
Sep 10 12:31:00.362240 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.362169    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mfltw"]
Sep 10 12:31:00.362358 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.362313    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:00.364891 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.364726    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 12:31:00.364891 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.364732    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 12:31:00.364891 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.364786    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sgmkk\""
Sep 10 12:31:00.384233 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.384198    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5qwhw"]
Sep 10 12:31:00.384357 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.384271    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mfltw"]
Sep 10 12:31:00.384405 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.384387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mfltw"
Sep 10 12:31:00.386916 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.386895    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zj5jx\""
Sep 10 12:31:00.387012 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.386948    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 12:31:00.387256 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.387240    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 12:31:00.387256 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.387249    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 12:31:00.518065 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.518026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r6nm\" (UniqueName: \"kubernetes.io/projected/1bf1ff93-c28e-404e-b894-b82f73484d09-kube-api-access-8r6nm\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:00.518233 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.518092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:00.518233 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.518135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf1ff93-c28e-404e-b894-b82f73484d09-tmp-dir\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:00.518233 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.518212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf1ff93-c28e-404e-b894-b82f73484d09-config-volume\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:00.518362 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.518250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2576131-0632-4603-a98c-e47cd0c73e69-cert\") pod \"ingress-canary-mfltw\" (UID: \"e2576131-0632-4603-a98c-e47cd0c73e69\") " pod="openshift-ingress-canary/ingress-canary-mfltw"
Sep 10 12:31:00.518362 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.518279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g47cn\" (UniqueName: \"kubernetes.io/projected/e2576131-0632-4603-a98c-e47cd0c73e69-kube-api-access-g47cn\") pod \"ingress-canary-mfltw\" (UID: \"e2576131-0632-4603-a98c-e47cd0c73e69\") " pod="openshift-ingress-canary/ingress-canary-mfltw"
Sep 10 12:31:00.619427 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.619346    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:00.619427 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.619388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf1ff93-c28e-404e-b894-b82f73484d09-tmp-dir\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:00.619673 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:00.619502    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:31:00.619673 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.619519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf1ff93-c28e-404e-b894-b82f73484d09-config-volume\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:00.619673 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:00.619577    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls podName:1bf1ff93-c28e-404e-b894-b82f73484d09 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:01.119545518 +0000 UTC m=+35.044457018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls") pod "dns-default-5qwhw" (UID: "1bf1ff93-c28e-404e-b894-b82f73484d09") : secret "dns-default-metrics-tls" not found
Sep 10 12:31:00.619673 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.619612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2576131-0632-4603-a98c-e47cd0c73e69-cert\") pod \"ingress-canary-mfltw\" (UID: \"e2576131-0632-4603-a98c-e47cd0c73e69\") " pod="openshift-ingress-canary/ingress-canary-mfltw"
Sep 10 12:31:00.619673 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.619643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g47cn\" (UniqueName: \"kubernetes.io/projected/e2576131-0632-4603-a98c-e47cd0c73e69-kube-api-access-g47cn\") pod \"ingress-canary-mfltw\" (UID: \"e2576131-0632-4603-a98c-e47cd0c73e69\") " pod="openshift-ingress-canary/ingress-canary-mfltw"
Sep 10 12:31:00.619673 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.619661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r6nm\" (UniqueName: \"kubernetes.io/projected/1bf1ff93-c28e-404e-b894-b82f73484d09-kube-api-access-8r6nm\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:00.619673 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.619663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bf1ff93-c28e-404e-b894-b82f73484d09-tmp-dir\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:00.620069 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.620048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf1ff93-c28e-404e-b894-b82f73484d09-config-volume\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:00.623629 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.623603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e2576131-0632-4603-a98c-e47cd0c73e69-cert\") pod \"ingress-canary-mfltw\" (UID: \"e2576131-0632-4603-a98c-e47cd0c73e69\") " pod="openshift-ingress-canary/ingress-canary-mfltw"
Sep 10 12:31:00.628159 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.628133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r6nm\" (UniqueName: \"kubernetes.io/projected/1bf1ff93-c28e-404e-b894-b82f73484d09-kube-api-access-8r6nm\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:00.628245 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.628217    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g47cn\" (UniqueName: \"kubernetes.io/projected/e2576131-0632-4603-a98c-e47cd0c73e69-kube-api-access-g47cn\") pod \"ingress-canary-mfltw\" (UID: \"e2576131-0632-4603-a98c-e47cd0c73e69\") " pod="openshift-ingress-canary/ingress-canary-mfltw"
Sep 10 12:31:00.692449 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.692428    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mfltw"
Sep 10 12:31:00.697944 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.697919    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:31:00.698117 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.698106    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:31:00.700408 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.700388    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 12:31:00.700508 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.700388    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 12:31:00.700508 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.700392    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sz48d\""
Sep 10 12:31:00.888363 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:00.888287    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntbv6" event={"ID":"688eff1f-bb3f-4223-9721-9c8898af37e8","Type":"ContainerStarted","Data":"3469c95a3457ec90e7e865564e59e48fb08ff49a2b0b268b20f10c103c9fd3fc"}
Sep 10 12:31:01.012716 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:01.012685    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mfltw"]
Sep 10 12:31:01.016234 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:31:01.016206    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2576131_0632_4603_a98c_e47cd0c73e69.slice/crio-1c1e0fdebcc6213981c98fe877c380e83ac98dac9393e6a896743f65e4d57110 WatchSource:0}: Error finding container 1c1e0fdebcc6213981c98fe877c380e83ac98dac9393e6a896743f65e4d57110: Status 404 returned error can't find the container with id 1c1e0fdebcc6213981c98fe877c380e83ac98dac9393e6a896743f65e4d57110
Sep 10 12:31:01.124041 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:01.124008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:01.124198 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:01.124175    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:31:01.124254 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:01.124244    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls podName:1bf1ff93-c28e-404e-b894-b82f73484d09 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:02.124228847 +0000 UTC m=+36.049140314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls") pod "dns-default-5qwhw" (UID: "1bf1ff93-c28e-404e-b894-b82f73484d09") : secret "dns-default-metrics-tls" not found
Sep 10 12:31:01.698654 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:01.698618    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:31:01.702889 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:01.702869    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 12:31:01.703728 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:01.703711    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ll527\""
Sep 10 12:31:01.703802 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:01.703749    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 12:31:01.892486 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:01.892400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mfltw" event={"ID":"e2576131-0632-4603-a98c-e47cd0c73e69","Type":"ContainerStarted","Data":"1c1e0fdebcc6213981c98fe877c380e83ac98dac9393e6a896743f65e4d57110"}
Sep 10 12:31:01.895356 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:01.895326    2563 generic.go:356] "Generic (PLEG): container finished" podID="688eff1f-bb3f-4223-9721-9c8898af37e8" containerID="3469c95a3457ec90e7e865564e59e48fb08ff49a2b0b268b20f10c103c9fd3fc" exitCode=0
Sep 10 12:31:01.895500 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:01.895394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntbv6" event={"ID":"688eff1f-bb3f-4223-9721-9c8898af37e8","Type":"ContainerDied","Data":"3469c95a3457ec90e7e865564e59e48fb08ff49a2b0b268b20f10c103c9fd3fc"}
Sep 10 12:31:02.133313 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:02.133284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:02.133487 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:02.133462    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:31:02.133567 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:02.133545    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls podName:1bf1ff93-c28e-404e-b894-b82f73484d09 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:04.133524494 +0000 UTC m=+38.058435980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls") pod "dns-default-5qwhw" (UID: "1bf1ff93-c28e-404e-b894-b82f73484d09") : secret "dns-default-metrics-tls" not found
Sep 10 12:31:02.902048 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:02.902006    2563 generic.go:356] "Generic (PLEG): container finished" podID="688eff1f-bb3f-4223-9721-9c8898af37e8" containerID="b503e32e002d6561a46140aa75ef4342b07f9679a5196014b356dfb7fe1ea34d" exitCode=0
Sep 10 12:31:02.902634 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:02.902079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntbv6" event={"ID":"688eff1f-bb3f-4223-9721-9c8898af37e8","Type":"ContainerDied","Data":"b503e32e002d6561a46140aa75ef4342b07f9679a5196014b356dfb7fe1ea34d"}
Sep 10 12:31:03.906476 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:03.906395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mfltw" event={"ID":"e2576131-0632-4603-a98c-e47cd0c73e69","Type":"ContainerStarted","Data":"c36e0f54f1d0ea8866ae892c6e52525f2588d426f27b746008f5345d54683628"}
Sep 10 12:31:03.909675 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:03.909651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ntbv6" event={"ID":"688eff1f-bb3f-4223-9721-9c8898af37e8","Type":"ContainerStarted","Data":"3b4de84dfa5cdde9aa0aeb5cb74884a8e30dd5807ef8b35aa4119ddbcf75a48c"}
Sep 10 12:31:03.923758 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:03.923712    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mfltw" podStartSLOduration=1.3333821860000001 podStartE2EDuration="3.923698003s" totalImagesPullingTime="2.590315817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:03.922179724 +0000 UTC m=+37.847091234" watchObservedRunningTime="2026-09-10 12:31:03.923698003 +0000 UTC m=+37.848609490"
Sep 10 12:31:03.945746 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:03.943850    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ntbv6" podStartSLOduration=9.217366208 podStartE2EDuration="37.943830744s" totalImagesPullingTime="28.726464536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:03.942103935 +0000 UTC m=+37.867015422" watchObservedRunningTime="2026-09-10 12:31:03.943830744 +0000 UTC m=+37.868742234"
Sep 10 12:31:04.148843 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:04.148807    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:04.149004 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:04.148942    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:31:04.149004 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:04.148998    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls podName:1bf1ff93-c28e-404e-b894-b82f73484d09 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:08.148983769 +0000 UTC m=+42.073895235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls") pod "dns-default-5qwhw" (UID: "1bf1ff93-c28e-404e-b894-b82f73484d09") : secret "dns-default-metrics-tls" not found
Sep 10 12:31:08.176322 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:08.176284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:08.176722 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:08.176409    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:31:08.176722 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:08.176462    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls podName:1bf1ff93-c28e-404e-b894-b82f73484d09 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:16.176449343 +0000 UTC m=+50.101360823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls") pod "dns-default-5qwhw" (UID: "1bf1ff93-c28e-404e-b894-b82f73484d09") : secret "dns-default-metrics-tls" not found
Sep 10 12:31:16.236530 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:16.236490    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:16.236962 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:16.236618    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:31:16.236962 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:16.236669    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls podName:1bf1ff93-c28e-404e-b894-b82f73484d09 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:32.236656408 +0000 UTC m=+66.161567873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls") pod "dns-default-5qwhw" (UID: "1bf1ff93-c28e-404e-b894-b82f73484d09") : secret "dns-default-metrics-tls" not found
Sep 10 12:31:25.883663 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:25.883637    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dtnsr"
Sep 10 12:31:32.247928 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.247886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:31:32.248405 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.247963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:31:32.248405 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:32.248046    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 12:31:32.248405 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:32.248111    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls podName:1bf1ff93-c28e-404e-b894-b82f73484d09 nodeName:}" failed. No retries permitted until 2026-09-10 12:32:04.248095742 +0000 UTC m=+98.173007207 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls") pod "dns-default-5qwhw" (UID: "1bf1ff93-c28e-404e-b894-b82f73484d09") : secret "dns-default-metrics-tls" not found
Sep 10 12:31:32.250866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.250850    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 12:31:32.258834 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:32.258818    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 12:31:32.258877 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:32.258866    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs podName:a04476a3-6d83-40ff-879c-c9a5b620ef44 nodeName:}" failed. No retries permitted until 2026-09-10 12:32:36.258853007 +0000 UTC m=+130.183764473 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs") pod "network-metrics-daemon-vj4n4" (UID: "a04476a3-6d83-40ff-879c-c9a5b620ef44") : secret "metrics-daemon-secret" not found
Sep 10 12:31:32.348905 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.348871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:31:32.348969 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.348917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhmq\" (UniqueName: \"kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq\") pod \"network-check-target-jz9tr\" (UID: \"911473b1-59c6-4916-a2d6-2dca448752a2\") " pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:31:32.352070 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.352050    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 12:31:32.352111 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.352092    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 12:31:32.361710 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.361686    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 12:31:32.362285 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.362268    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4be22799-8b7d-4e18-867f-34a3185af15a-original-pull-secret\") pod \"global-pull-secret-syncer-jfpm9\" (UID: \"4be22799-8b7d-4e18-867f-34a3185af15a\") " pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:31:32.371932 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.371913    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rhmq\" (UniqueName: \"kubernetes.io/projected/911473b1-59c6-4916-a2d6-2dca448752a2-kube-api-access-5rhmq\") pod \"network-check-target-jz9tr\" (UID: \"911473b1-59c6-4916-a2d6-2dca448752a2\") " pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:31:32.512376 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.512284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jfpm9"
Sep 10 12:31:32.609241 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.609206    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ll527\""
Sep 10 12:31:32.617112 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.617088    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:31:32.781984 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.781916    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jfpm9"]
Sep 10 12:31:32.785063 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:31:32.785032    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be22799_8b7d_4e18_867f_34a3185af15a.slice/crio-9ae8ae91d48362b204c9f9ec08687baa440147d16cb2828fc89f1d1c16258d6a WatchSource:0}: Error finding container 9ae8ae91d48362b204c9f9ec08687baa440147d16cb2828fc89f1d1c16258d6a: Status 404 returned error can't find the container with id 9ae8ae91d48362b204c9f9ec08687baa440147d16cb2828fc89f1d1c16258d6a
Sep 10 12:31:32.883947 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.883912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jz9tr"]
Sep 10 12:31:32.889103 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:31:32.889076    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod911473b1_59c6_4916_a2d6_2dca448752a2.slice/crio-7c2a2c943b8050534db7c145c020101e346bdd0d1f2aff8ec2e17f83b6b11f68 WatchSource:0}: Error finding container 7c2a2c943b8050534db7c145c020101e346bdd0d1f2aff8ec2e17f83b6b11f68: Status 404 returned error can't find the container with id 7c2a2c943b8050534db7c145c020101e346bdd0d1f2aff8ec2e17f83b6b11f68
Sep 10 12:31:32.982455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.982420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jfpm9" event={"ID":"4be22799-8b7d-4e18-867f-34a3185af15a","Type":"ContainerStarted","Data":"9ae8ae91d48362b204c9f9ec08687baa440147d16cb2828fc89f1d1c16258d6a"}
Sep 10 12:31:32.983483 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:32.983459    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jz9tr" event={"ID":"911473b1-59c6-4916-a2d6-2dca448752a2","Type":"ContainerStarted","Data":"7c2a2c943b8050534db7c145c020101e346bdd0d1f2aff8ec2e17f83b6b11f68"}
Sep 10 12:31:37.999123 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:37.999083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jz9tr" event={"ID":"911473b1-59c6-4916-a2d6-2dca448752a2","Type":"ContainerStarted","Data":"0f72356c21c0df76ddf68939ed0cabce819694b46b0f637f741159fa7a7bfca1"}
Sep 10 12:31:37.999657 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:37.999221    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:31:38.000580 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:38.000543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jfpm9" event={"ID":"4be22799-8b7d-4e18-867f-34a3185af15a","Type":"ContainerStarted","Data":"afe82baca33b17d04f426e298643c7b8923b9a7bd841dfe91bbf2440894c3634"}
Sep 10 12:31:38.018079 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:38.018016    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jz9tr" podStartSLOduration=67.800404502 podStartE2EDuration="1m12.017998149s" totalImagesPullingTime="4.217593647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:38.016280485 +0000 UTC m=+71.941191989" watchObservedRunningTime="2026-09-10 12:31:38.017998149 +0000 UTC m=+71.942909639"
Sep 10 12:31:38.032866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:38.032814    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jfpm9" podStartSLOduration=67.705043011 podStartE2EDuration="1m12.032798274s" totalImagesPullingTime="4.327755263s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:38.032659489 +0000 UTC m=+71.957570976" watchObservedRunningTime="2026-09-10 12:31:38.032798274 +0000 UTC m=+71.957709763"
Sep 10 12:31:41.452908 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.452864    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"]
Sep 10 12:31:41.454854 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.454838    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:41.457282 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.457258    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 12:31:41.457406 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.457292    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 12:31:41.458088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.458072    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5vxz4\""
Sep 10 12:31:41.459408 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.459390    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 12:31:41.459491 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.459433    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 12:31:41.470693 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.470674    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"]
Sep 10 12:31:41.509190 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.509163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:41.509347 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.509211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jwml\" (UniqueName: \"kubernetes.io/projected/b897d389-1590-4c48-9504-24c834416b86-kube-api-access-2jwml\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:41.509347 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.509230    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b897d389-1590-4c48-9504-24c834416b86-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:41.558487 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.558458    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"]
Sep 10 12:31:41.560179 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.560162    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"
Sep 10 12:31:41.563938 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.563915    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:31:41.564050 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.563998    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 10 12:31:41.564215 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.564201    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 10 12:31:41.564445 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.564424    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ctbpn\""
Sep 10 12:31:41.567324 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.567301    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 10 12:31:41.576594 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.576573    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"]
Sep 10 12:31:41.610103 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.610075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b897d389-1590-4c48-9504-24c834416b86-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:41.610230 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.610124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:41.610230 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.610157    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49d13c55-3c7d-4df7-8593-ae68b8b31f77-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-584zz\" (UID: \"49d13c55-3c7d-4df7-8593-ae68b8b31f77\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"
Sep 10 12:31:41.610230 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.610174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxp66\" (UniqueName: \"kubernetes.io/projected/49d13c55-3c7d-4df7-8593-ae68b8b31f77-kube-api-access-rxp66\") pod \"kube-storage-version-migrator-operator-8d7dd4675-584zz\" (UID: \"49d13c55-3c7d-4df7-8593-ae68b8b31f77\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"
Sep 10 12:31:41.610230 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.610203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jwml\" (UniqueName: \"kubernetes.io/projected/b897d389-1590-4c48-9504-24c834416b86-kube-api-access-2jwml\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:41.610230 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.610221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49d13c55-3c7d-4df7-8593-ae68b8b31f77-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-584zz\" (UID: \"49d13c55-3c7d-4df7-8593-ae68b8b31f77\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"
Sep 10 12:31:41.610472 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:41.610377    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 12:31:41.610472 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:41.610462    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls podName:b897d389-1590-4c48-9504-24c834416b86 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:42.110430987 +0000 UTC m=+76.035342478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-nbc4r" (UID: "b897d389-1590-4c48-9504-24c834416b86") : secret "cluster-monitoring-operator-tls" not found
Sep 10 12:31:41.610744 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.610725    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b897d389-1590-4c48-9504-24c834416b86-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:41.630362 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.630333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jwml\" (UniqueName: \"kubernetes.io/projected/b897d389-1590-4c48-9504-24c834416b86-kube-api-access-2jwml\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:41.669321 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.669286    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58b575c474-nfrb2"]
Sep 10 12:31:41.671326 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.671311    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.677307 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.677279    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"]
Sep 10 12:31:41.680362 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.680335    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"
Sep 10 12:31:41.681247 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.681222    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 12:31:41.682610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.682485    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 12:31:41.682610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.682599    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 12:31:41.682920 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.682904    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t2rx6\""
Sep 10 12:31:41.693091 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.693067    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 12:31:41.693186 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.693173    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 12:31:41.693243 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.693175    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 12:31:41.693300 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.693227    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:31:41.693458 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.693444    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zq48g\""
Sep 10 12:31:41.693979 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.693960    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"]
Sep 10 12:31:41.697599 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.697581    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58b575c474-nfrb2"]
Sep 10 12:31:41.699501 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.699478    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 12:31:41.710700 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.710644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxp66\" (UniqueName: \"kubernetes.io/projected/49d13c55-3c7d-4df7-8593-ae68b8b31f77-kube-api-access-rxp66\") pod \"kube-storage-version-migrator-operator-8d7dd4675-584zz\" (UID: \"49d13c55-3c7d-4df7-8593-ae68b8b31f77\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"
Sep 10 12:31:41.710700 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.710679    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0de14be1-91b2-4b99-9a58-d73f2f699570-serving-cert\") pod \"service-ca-operator-859f94c4b-fn2vb\" (UID: \"0de14be1-91b2-4b99-9a58-d73f2f699570\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"
Sep 10 12:31:41.710829 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.710705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49d13c55-3c7d-4df7-8593-ae68b8b31f77-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-584zz\" (UID: \"49d13c55-3c7d-4df7-8593-ae68b8b31f77\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"
Sep 10 12:31:41.710829 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.710721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2456a6c7-1269-4dca-a837-1619420c6fd4-ca-trust-extracted\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.710829 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.710744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2456a6c7-1269-4dca-a837-1619420c6fd4-image-registry-private-configuration\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.710977 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.710868    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2456a6c7-1269-4dca-a837-1619420c6fd4-installation-pull-secrets\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.710977 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.710909    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2456a6c7-1269-4dca-a837-1619420c6fd4-trusted-ca\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.710977 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.710944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-bound-sa-token\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.711126 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.711011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-certificates\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.711126 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.711038    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xdnt\" (UniqueName: \"kubernetes.io/projected/0de14be1-91b2-4b99-9a58-d73f2f699570-kube-api-access-8xdnt\") pod \"service-ca-operator-859f94c4b-fn2vb\" (UID: \"0de14be1-91b2-4b99-9a58-d73f2f699570\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"
Sep 10 12:31:41.711126 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.711076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.711126 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.711099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sk6h\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-kube-api-access-8sk6h\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.711126 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.711122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0de14be1-91b2-4b99-9a58-d73f2f699570-config\") pod \"service-ca-operator-859f94c4b-fn2vb\" (UID: \"0de14be1-91b2-4b99-9a58-d73f2f699570\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"
Sep 10 12:31:41.711365 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.711183    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49d13c55-3c7d-4df7-8593-ae68b8b31f77-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-584zz\" (UID: \"49d13c55-3c7d-4df7-8593-ae68b8b31f77\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"
Sep 10 12:31:41.711709 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.711689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/49d13c55-3c7d-4df7-8593-ae68b8b31f77-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-584zz\" (UID: \"49d13c55-3c7d-4df7-8593-ae68b8b31f77\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"
Sep 10 12:31:41.713046 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.713029    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/49d13c55-3c7d-4df7-8593-ae68b8b31f77-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-584zz\" (UID: \"49d13c55-3c7d-4df7-8593-ae68b8b31f77\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"
Sep 10 12:31:41.730914 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.730893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxp66\" (UniqueName: \"kubernetes.io/projected/49d13c55-3c7d-4df7-8593-ae68b8b31f77-kube-api-access-rxp66\") pod \"kube-storage-version-migrator-operator-8d7dd4675-584zz\" (UID: \"49d13c55-3c7d-4df7-8593-ae68b8b31f77\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"
Sep 10 12:31:41.812091 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.812051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2456a6c7-1269-4dca-a837-1619420c6fd4-ca-trust-extracted\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.812091 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.812092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2456a6c7-1269-4dca-a837-1619420c6fd4-image-registry-private-configuration\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.812310 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.812110    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2456a6c7-1269-4dca-a837-1619420c6fd4-installation-pull-secrets\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.812310 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.812129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2456a6c7-1269-4dca-a837-1619420c6fd4-trusted-ca\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.812310 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.812152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-bound-sa-token\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.812310 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.812206    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-certificates\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.812310 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.812229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xdnt\" (UniqueName: \"kubernetes.io/projected/0de14be1-91b2-4b99-9a58-d73f2f699570-kube-api-access-8xdnt\") pod \"service-ca-operator-859f94c4b-fn2vb\" (UID: \"0de14be1-91b2-4b99-9a58-d73f2f699570\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"
Sep 10 12:31:41.812310 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.812261    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.812310 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.812283    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sk6h\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-kube-api-access-8sk6h\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.812310 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.812306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0de14be1-91b2-4b99-9a58-d73f2f699570-config\") pod \"service-ca-operator-859f94c4b-fn2vb\" (UID: \"0de14be1-91b2-4b99-9a58-d73f2f699570\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"
Sep 10 12:31:41.812702 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.812355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0de14be1-91b2-4b99-9a58-d73f2f699570-serving-cert\") pod \"service-ca-operator-859f94c4b-fn2vb\" (UID: \"0de14be1-91b2-4b99-9a58-d73f2f699570\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"
Sep 10 12:31:41.812702 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.812600    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2456a6c7-1269-4dca-a837-1619420c6fd4-ca-trust-extracted\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.812702 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:41.812605    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 12:31:41.812702 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:41.812637    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58b575c474-nfrb2: secret "image-registry-tls" not found
Sep 10 12:31:41.812702 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:41.812703    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls podName:2456a6c7-1269-4dca-a837-1619420c6fd4 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:42.312683059 +0000 UTC m=+76.237594535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls") pod "image-registry-58b575c474-nfrb2" (UID: "2456a6c7-1269-4dca-a837-1619420c6fd4") : secret "image-registry-tls" not found
Sep 10 12:31:41.813285 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.813259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0de14be1-91b2-4b99-9a58-d73f2f699570-config\") pod \"service-ca-operator-859f94c4b-fn2vb\" (UID: \"0de14be1-91b2-4b99-9a58-d73f2f699570\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"
Sep 10 12:31:41.813359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.813312    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-certificates\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.813359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.813334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2456a6c7-1269-4dca-a837-1619420c6fd4-trusted-ca\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.814708 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.814685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2456a6c7-1269-4dca-a837-1619420c6fd4-installation-pull-secrets\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.815025 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.815002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0de14be1-91b2-4b99-9a58-d73f2f699570-serving-cert\") pod \"service-ca-operator-859f94c4b-fn2vb\" (UID: \"0de14be1-91b2-4b99-9a58-d73f2f699570\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"
Sep 10 12:31:41.815131 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.815112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2456a6c7-1269-4dca-a837-1619420c6fd4-image-registry-private-configuration\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.821253 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.821227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sk6h\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-kube-api-access-8sk6h\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.821634 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.821616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-bound-sa-token\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:41.822075 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.822057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xdnt\" (UniqueName: \"kubernetes.io/projected/0de14be1-91b2-4b99-9a58-d73f2f699570-kube-api-access-8xdnt\") pod \"service-ca-operator-859f94c4b-fn2vb\" (UID: \"0de14be1-91b2-4b99-9a58-d73f2f699570\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"
Sep 10 12:31:41.869050 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.869016    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"
Sep 10 12:31:41.989260 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:41.989151    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"
Sep 10 12:31:42.115765 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:42.115729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:42.115904 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:42.115869    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 12:31:42.115943 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:42.115935    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls podName:b897d389-1590-4c48-9504-24c834416b86 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:43.115918333 +0000 UTC m=+77.040829803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-nbc4r" (UID: "b897d389-1590-4c48-9504-24c834416b86") : secret "cluster-monitoring-operator-tls" not found
Sep 10 12:31:42.134505 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:42.134475    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz"]
Sep 10 12:31:42.138743 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:31:42.138713    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d13c55_3c7d_4df7_8593_ae68b8b31f77.slice/crio-813dfcf34ea8d0f0e2b86d26c0061bf964883e4d6f04a05d80983b77778a4ad7 WatchSource:0}: Error finding container 813dfcf34ea8d0f0e2b86d26c0061bf964883e4d6f04a05d80983b77778a4ad7: Status 404 returned error can't find the container with id 813dfcf34ea8d0f0e2b86d26c0061bf964883e4d6f04a05d80983b77778a4ad7
Sep 10 12:31:42.256141 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:42.256066    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb"]
Sep 10 12:31:42.259268 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:31:42.259239    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de14be1_91b2_4b99_9a58_d73f2f699570.slice/crio-20155ecad732a984c334ecab01015dadd98c06b8f96c6f386c12eb9e95208cbb WatchSource:0}: Error finding container 20155ecad732a984c334ecab01015dadd98c06b8f96c6f386c12eb9e95208cbb: Status 404 returned error can't find the container with id 20155ecad732a984c334ecab01015dadd98c06b8f96c6f386c12eb9e95208cbb
Sep 10 12:31:42.318331 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:42.318295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:42.318478 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:42.318439    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 12:31:42.318478 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:42.318458    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58b575c474-nfrb2: secret "image-registry-tls" not found
Sep 10 12:31:42.318544 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:42.318512    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls podName:2456a6c7-1269-4dca-a837-1619420c6fd4 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:43.318494548 +0000 UTC m=+77.243406017 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls") pod "image-registry-58b575c474-nfrb2" (UID: "2456a6c7-1269-4dca-a837-1619420c6fd4") : secret "image-registry-tls" not found
Sep 10 12:31:43.015369 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:43.015326    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz" event={"ID":"49d13c55-3c7d-4df7-8593-ae68b8b31f77","Type":"ContainerStarted","Data":"813dfcf34ea8d0f0e2b86d26c0061bf964883e4d6f04a05d80983b77778a4ad7"}
Sep 10 12:31:43.016674 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:43.016636    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb" event={"ID":"0de14be1-91b2-4b99-9a58-d73f2f699570","Type":"ContainerStarted","Data":"20155ecad732a984c334ecab01015dadd98c06b8f96c6f386c12eb9e95208cbb"}
Sep 10 12:31:43.125201 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:43.125155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:43.125392 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:43.125337    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 12:31:43.125458 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:43.125436    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls podName:b897d389-1590-4c48-9504-24c834416b86 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:45.125413002 +0000 UTC m=+79.050324481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-nbc4r" (UID: "b897d389-1590-4c48-9504-24c834416b86") : secret "cluster-monitoring-operator-tls" not found
Sep 10 12:31:43.327004 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:43.326907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:43.327170 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:43.327041    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 12:31:43.327170 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:43.327053    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58b575c474-nfrb2: secret "image-registry-tls" not found
Sep 10 12:31:43.327170 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:43.327103    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls podName:2456a6c7-1269-4dca-a837-1619420c6fd4 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:45.327088559 +0000 UTC m=+79.252000025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls") pod "image-registry-58b575c474-nfrb2" (UID: "2456a6c7-1269-4dca-a837-1619420c6fd4") : secret "image-registry-tls" not found
Sep 10 12:31:45.022840 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.022797    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz" event={"ID":"49d13c55-3c7d-4df7-8593-ae68b8b31f77","Type":"ContainerStarted","Data":"39c693ba189c4ddf9450021ea2ae4138c60ba454e5c15d174b3e39c0ebda2d4c"}
Sep 10 12:31:45.024291 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.024262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb" event={"ID":"0de14be1-91b2-4b99-9a58-d73f2f699570","Type":"ContainerStarted","Data":"b2cc915f675e60768c46d6f2d7f35539d987ba7784e28d469dcdd41cdda79de1"}
Sep 10 12:31:45.043988 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.043931    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz" podStartSLOduration=2.193289201 podStartE2EDuration="4.043914462s" totalImagesPullingTime="1.850625261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:45.042538065 +0000 UTC m=+78.967449552" watchObservedRunningTime="2026-09-10 12:31:45.043914462 +0000 UTC m=+78.968825984"
Sep 10 12:31:45.069737 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.069679    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb" podStartSLOduration=1.7839488650000002 podStartE2EDuration="4.069658317s" totalImagesPullingTime="2.285709452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:45.068285166 +0000 UTC m=+78.993196654" watchObservedRunningTime="2026-09-10 12:31:45.069658317 +0000 UTC m=+78.994569807"
Sep 10 12:31:45.142336 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.142292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:45.142514 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:45.142432    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 12:31:45.142514 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:45.142506    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls podName:b897d389-1590-4c48-9504-24c834416b86 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:49.14248422 +0000 UTC m=+83.067395707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-nbc4r" (UID: "b897d389-1590-4c48-9504-24c834416b86") : secret "cluster-monitoring-operator-tls" not found
Sep 10 12:31:45.345036 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.344932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:45.345214 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:45.345083    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 12:31:45.345214 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:45.345105    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58b575c474-nfrb2: secret "image-registry-tls" not found
Sep 10 12:31:45.345214 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:45.345171    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls podName:2456a6c7-1269-4dca-a837-1619420c6fd4 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:49.345155026 +0000 UTC m=+83.270066509 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls") pod "image-registry-58b575c474-nfrb2" (UID: "2456a6c7-1269-4dca-a837-1619420c6fd4") : secret "image-registry-tls" not found
Sep 10 12:31:45.653705 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.653629    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tqkdr"]
Sep 10 12:31:45.656018 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.656002    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:45.661672 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.661636    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 12:31:45.661793 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.661776    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qvkkn\""
Sep 10 12:31:45.662574 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.662539    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 12:31:45.662676 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.662619    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 12:31:45.662865 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.662850    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 12:31:45.668059 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.668037    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tqkdr"]
Sep 10 12:31:45.747088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.747054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzfl\" (UniqueName: \"kubernetes.io/projected/7a8ce703-8992-44dc-a56c-1313710a627d-kube-api-access-hbzfl\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:45.747088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.747089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:45.747271 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.747128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8ce703-8992-44dc-a56c-1313710a627d-crio-socket\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:45.747271 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.747170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:45.747271 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.747187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a8ce703-8992-44dc-a56c-1313710a627d-data-volume\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:45.848472 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.848431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:45.848472 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.848473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a8ce703-8992-44dc-a56c-1313710a627d-data-volume\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:45.848721 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.848501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzfl\" (UniqueName: \"kubernetes.io/projected/7a8ce703-8992-44dc-a56c-1313710a627d-kube-api-access-hbzfl\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:45.848721 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.848525    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:45.848721 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.848589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8ce703-8992-44dc-a56c-1313710a627d-crio-socket\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:45.848721 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:45.848622    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:45.848721 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:45.848661    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 12:31:45.848721 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:45.848703    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls podName:7a8ce703-8992-44dc-a56c-1313710a627d nodeName:}" failed. No retries permitted until 2026-09-10 12:31:46.348682503 +0000 UTC m=+80.273593969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqkdr" (UID: "7a8ce703-8992-44dc-a56c-1313710a627d") : secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:45.848721 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.848717    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7a8ce703-8992-44dc-a56c-1313710a627d-crio-socket\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:45.848959 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:45.848728    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm podName:7a8ce703-8992-44dc-a56c-1313710a627d nodeName:}" failed. No retries permitted until 2026-09-10 12:31:46.348710512 +0000 UTC m=+80.273622003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tqkdr" (UID: "7a8ce703-8992-44dc-a56c-1313710a627d") : configmap "kube-rbac-proxy" not found
Sep 10 12:31:45.848959 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.848836    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7a8ce703-8992-44dc-a56c-1313710a627d-data-volume\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:45.865231 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:45.865199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzfl\" (UniqueName: \"kubernetes.io/projected/7a8ce703-8992-44dc-a56c-1313710a627d-kube-api-access-hbzfl\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:46.351905 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:46.351848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:46.352352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:46.351930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:46.352352 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:46.352002    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:46.352352 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:46.352017    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 12:31:46.352352 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:46.352069    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls podName:7a8ce703-8992-44dc-a56c-1313710a627d nodeName:}" failed. No retries permitted until 2026-09-10 12:31:47.352054964 +0000 UTC m=+81.276966430 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqkdr" (UID: "7a8ce703-8992-44dc-a56c-1313710a627d") : secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:46.352352 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:46.352085    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm podName:7a8ce703-8992-44dc-a56c-1313710a627d nodeName:}" failed. No retries permitted until 2026-09-10 12:31:47.3520777 +0000 UTC m=+81.276989167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tqkdr" (UID: "7a8ce703-8992-44dc-a56c-1313710a627d") : configmap "kube-rbac-proxy" not found
Sep 10 12:31:47.359432 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:47.359396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:47.359873 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:47.359449    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:47.359873 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:47.359542    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 12:31:47.359873 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:47.359594    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:47.359873 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:47.359633    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm podName:7a8ce703-8992-44dc-a56c-1313710a627d nodeName:}" failed. No retries permitted until 2026-09-10 12:31:49.359620182 +0000 UTC m=+83.284531661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tqkdr" (UID: "7a8ce703-8992-44dc-a56c-1313710a627d") : configmap "kube-rbac-proxy" not found
Sep 10 12:31:47.359873 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:47.359659    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls podName:7a8ce703-8992-44dc-a56c-1313710a627d nodeName:}" failed. No retries permitted until 2026-09-10 12:31:49.359640598 +0000 UTC m=+83.284552078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqkdr" (UID: "7a8ce703-8992-44dc-a56c-1313710a627d") : secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:47.882643 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:47.882611    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-vsmfh"]
Sep 10 12:31:47.885285 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:47.885268    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-vsmfh"
Sep 10 12:31:47.887850 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:47.887823    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 12:31:47.888644 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:47.888625    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 12:31:47.888764 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:47.888642    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4qw29\""
Sep 10 12:31:47.888764 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:47.888674    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 12:31:47.888764 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:47.888646    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 12:31:47.896095 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:47.896074    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-vsmfh"]
Sep 10 12:31:47.965064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:47.965030    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/60e398b9-ee5e-4be6-983c-11a9eebc86fb-signing-key\") pod \"service-ca-75d99c484-vsmfh\" (UID: \"60e398b9-ee5e-4be6-983c-11a9eebc86fb\") " pod="openshift-service-ca/service-ca-75d99c484-vsmfh"
Sep 10 12:31:47.965064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:47.965069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpmvv\" (UniqueName: \"kubernetes.io/projected/60e398b9-ee5e-4be6-983c-11a9eebc86fb-kube-api-access-dpmvv\") pod \"service-ca-75d99c484-vsmfh\" (UID: \"60e398b9-ee5e-4be6-983c-11a9eebc86fb\") " pod="openshift-service-ca/service-ca-75d99c484-vsmfh"
Sep 10 12:31:47.965269 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:47.965107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/60e398b9-ee5e-4be6-983c-11a9eebc86fb-signing-cabundle\") pod \"service-ca-75d99c484-vsmfh\" (UID: \"60e398b9-ee5e-4be6-983c-11a9eebc86fb\") " pod="openshift-service-ca/service-ca-75d99c484-vsmfh"
Sep 10 12:31:48.066478 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:48.066436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/60e398b9-ee5e-4be6-983c-11a9eebc86fb-signing-key\") pod \"service-ca-75d99c484-vsmfh\" (UID: \"60e398b9-ee5e-4be6-983c-11a9eebc86fb\") " pod="openshift-service-ca/service-ca-75d99c484-vsmfh"
Sep 10 12:31:48.066478 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:48.066479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpmvv\" (UniqueName: \"kubernetes.io/projected/60e398b9-ee5e-4be6-983c-11a9eebc86fb-kube-api-access-dpmvv\") pod \"service-ca-75d99c484-vsmfh\" (UID: \"60e398b9-ee5e-4be6-983c-11a9eebc86fb\") " pod="openshift-service-ca/service-ca-75d99c484-vsmfh"
Sep 10 12:31:48.066697 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:48.066532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/60e398b9-ee5e-4be6-983c-11a9eebc86fb-signing-cabundle\") pod \"service-ca-75d99c484-vsmfh\" (UID: \"60e398b9-ee5e-4be6-983c-11a9eebc86fb\") " pod="openshift-service-ca/service-ca-75d99c484-vsmfh"
Sep 10 12:31:48.067235 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:48.067216    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/60e398b9-ee5e-4be6-983c-11a9eebc86fb-signing-cabundle\") pod \"service-ca-75d99c484-vsmfh\" (UID: \"60e398b9-ee5e-4be6-983c-11a9eebc86fb\") " pod="openshift-service-ca/service-ca-75d99c484-vsmfh"
Sep 10 12:31:48.068993 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:48.068975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/60e398b9-ee5e-4be6-983c-11a9eebc86fb-signing-key\") pod \"service-ca-75d99c484-vsmfh\" (UID: \"60e398b9-ee5e-4be6-983c-11a9eebc86fb\") " pod="openshift-service-ca/service-ca-75d99c484-vsmfh"
Sep 10 12:31:48.075032 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:48.075012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpmvv\" (UniqueName: \"kubernetes.io/projected/60e398b9-ee5e-4be6-983c-11a9eebc86fb-kube-api-access-dpmvv\") pod \"service-ca-75d99c484-vsmfh\" (UID: \"60e398b9-ee5e-4be6-983c-11a9eebc86fb\") " pod="openshift-service-ca/service-ca-75d99c484-vsmfh"
Sep 10 12:31:48.195130 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:48.195094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-vsmfh"
Sep 10 12:31:48.466345 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:48.466264    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-vsmfh"]
Sep 10 12:31:48.470031 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:31:48.470007    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e398b9_ee5e_4be6_983c_11a9eebc86fb.slice/crio-fd20543add25356aa46a92f6c0d96869160a3c7db83eba360c8eb4d36acad899 WatchSource:0}: Error finding container fd20543add25356aa46a92f6c0d96869160a3c7db83eba360c8eb4d36acad899: Status 404 returned error can't find the container with id fd20543add25356aa46a92f6c0d96869160a3c7db83eba360c8eb4d36acad899
Sep 10 12:31:48.539068 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:48.539043    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7fz9_a6e7308e-da70-4a2b-845a-51057e2f1f94/dns-node-resolver/0.log"
Sep 10 12:31:49.036230 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:49.036199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-vsmfh" event={"ID":"60e398b9-ee5e-4be6-983c-11a9eebc86fb","Type":"ContainerStarted","Data":"b6a3a34847ea6f47f7ba4c20f36c71756c19505356591716338aac71389f206d"}
Sep 10 12:31:49.036395 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:49.036243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-vsmfh" event={"ID":"60e398b9-ee5e-4be6-983c-11a9eebc86fb","Type":"ContainerStarted","Data":"fd20543add25356aa46a92f6c0d96869160a3c7db83eba360c8eb4d36acad899"}
Sep 10 12:31:49.057199 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:49.057145    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-vsmfh" podStartSLOduration=2.057126715 podStartE2EDuration="2.057126715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:49.057089112 +0000 UTC m=+82.982000600" watchObservedRunningTime="2026-09-10 12:31:49.057126715 +0000 UTC m=+82.982038203"
Sep 10 12:31:49.176454 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:49.176415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:49.176653 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:49.176581    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 12:31:49.176653 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:49.176644    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls podName:b897d389-1590-4c48-9504-24c834416b86 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:57.176628022 +0000 UTC m=+91.101539488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-nbc4r" (UID: "b897d389-1590-4c48-9504-24c834416b86") : secret "cluster-monitoring-operator-tls" not found
Sep 10 12:31:49.338599 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:49.338522    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n5899_c1e745f6-42ce-443e-8555-1572dbad7050/node-ca/0.log"
Sep 10 12:31:49.379077 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:49.379045    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:49.379237 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:49.379166    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:49.379237 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:49.379186    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 12:31:49.379237 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:49.379201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:49.379237 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:49.379209    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58b575c474-nfrb2: secret "image-registry-tls" not found
Sep 10 12:31:49.379428 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:49.379270    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 12:31:49.379428 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:49.379272    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls podName:2456a6c7-1269-4dca-a837-1619420c6fd4 nodeName:}" failed. No retries permitted until 2026-09-10 12:31:57.379251909 +0000 UTC m=+91.304163380 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls") pod "image-registry-58b575c474-nfrb2" (UID: "2456a6c7-1269-4dca-a837-1619420c6fd4") : secret "image-registry-tls" not found
Sep 10 12:31:49.379428 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:49.379315    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:49.379428 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:49.379323    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm podName:7a8ce703-8992-44dc-a56c-1313710a627d nodeName:}" failed. No retries permitted until 2026-09-10 12:31:53.37930794 +0000 UTC m=+87.304219409 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tqkdr" (UID: "7a8ce703-8992-44dc-a56c-1313710a627d") : configmap "kube-rbac-proxy" not found
Sep 10 12:31:49.379428 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:49.379388    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls podName:7a8ce703-8992-44dc-a56c-1313710a627d nodeName:}" failed. No retries permitted until 2026-09-10 12:31:53.379372137 +0000 UTC m=+87.304283616 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqkdr" (UID: "7a8ce703-8992-44dc-a56c-1313710a627d") : secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:49.939586 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:49.939537    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mfltw_e2576131-0632-4603-a98c-e47cd0c73e69/serve-healthcheck-canary/0.log"
Sep 10 12:31:50.943698 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:50.943667    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-584zz_49d13c55-3c7d-4df7-8593-ae68b8b31f77/kube-storage-version-migrator-operator/0.log"
Sep 10 12:31:53.413777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:53.413725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:53.414220 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:53.413794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:31:53.414220 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:53.413876    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 12:31:53.414220 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:53.413895    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:53.414220 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:53.413950    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls podName:7a8ce703-8992-44dc-a56c-1313710a627d nodeName:}" failed. No retries permitted until 2026-09-10 12:32:01.413935058 +0000 UTC m=+95.338846524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tqkdr" (UID: "7a8ce703-8992-44dc-a56c-1313710a627d") : secret "insights-runtime-extractor-tls" not found
Sep 10 12:31:53.414220 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:53.413965    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm podName:7a8ce703-8992-44dc-a56c-1313710a627d nodeName:}" failed. No retries permitted until 2026-09-10 12:32:01.413956108 +0000 UTC m=+95.338867574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tqkdr" (UID: "7a8ce703-8992-44dc-a56c-1313710a627d") : configmap "kube-rbac-proxy" not found
Sep 10 12:31:57.243744 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:57.243704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:31:57.244241 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:57.243886    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 12:31:57.244241 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:31:57.243981    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls podName:b897d389-1590-4c48-9504-24c834416b86 nodeName:}" failed. No retries permitted until 2026-09-10 12:32:13.243957365 +0000 UTC m=+107.168868845 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-nbc4r" (UID: "b897d389-1590-4c48-9504-24c834416b86") : secret "cluster-monitoring-operator-tls" not found
Sep 10 12:31:57.445587 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:57.445523    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:57.447918 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:57.447889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls\") pod \"image-registry-58b575c474-nfrb2\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") " pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:57.581972 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:57.581880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:57.856924 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:57.856841    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58b575c474-nfrb2"]
Sep 10 12:31:57.859887 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:31:57.859855    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2456a6c7_1269_4dca_a837_1619420c6fd4.slice/crio-a91621c5d43410465bfbd03dfd5a2dfb0b128255512515f86cf9118ff9e0dfa0 WatchSource:0}: Error finding container a91621c5d43410465bfbd03dfd5a2dfb0b128255512515f86cf9118ff9e0dfa0: Status 404 returned error can't find the container with id a91621c5d43410465bfbd03dfd5a2dfb0b128255512515f86cf9118ff9e0dfa0
Sep 10 12:31:58.071829 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:58.071791    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58b575c474-nfrb2" event={"ID":"2456a6c7-1269-4dca-a837-1619420c6fd4","Type":"ContainerStarted","Data":"509cbf0eb8c34455a949215a2addc5fac93f41c46f20460ad7236a19575c1979"}
Sep 10 12:31:58.071968 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:58.071851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58b575c474-nfrb2" event={"ID":"2456a6c7-1269-4dca-a837-1619420c6fd4","Type":"ContainerStarted","Data":"a91621c5d43410465bfbd03dfd5a2dfb0b128255512515f86cf9118ff9e0dfa0"}
Sep 10 12:31:58.071968 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:58.071880    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:31:58.100037 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:31:58.099976    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58b575c474-nfrb2" podStartSLOduration=17.099952362 podStartE2EDuration="17.099952362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:58.098666859 +0000 UTC m=+92.023578352" watchObservedRunningTime="2026-09-10 12:31:58.099952362 +0000 UTC m=+92.024863854"
Sep 10 12:32:01.479568 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:01.479518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:32:01.480043 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:01.479587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:32:01.480098 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:01.480050    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7a8ce703-8992-44dc-a56c-1313710a627d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:32:01.481940 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:01.481916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7a8ce703-8992-44dc-a56c-1313710a627d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tqkdr\" (UID: \"7a8ce703-8992-44dc-a56c-1313710a627d\") " pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:32:01.564874 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:01.564840    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tqkdr"
Sep 10 12:32:01.837675 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:01.837599    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tqkdr"]
Sep 10 12:32:01.840970 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:32:01.840946    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8ce703_8992_44dc_a56c_1313710a627d.slice/crio-1957dcccf4cb92ecb3ae12cd530b7a35e2bd37866fd97f5e9221fe0e5032cea1 WatchSource:0}: Error finding container 1957dcccf4cb92ecb3ae12cd530b7a35e2bd37866fd97f5e9221fe0e5032cea1: Status 404 returned error can't find the container with id 1957dcccf4cb92ecb3ae12cd530b7a35e2bd37866fd97f5e9221fe0e5032cea1
Sep 10 12:32:02.085265 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:02.085230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tqkdr" event={"ID":"7a8ce703-8992-44dc-a56c-1313710a627d","Type":"ContainerStarted","Data":"4600a9a8853f1e1e6b42f43c158d62c6a94b46f466e609b81669314f543a0552"}
Sep 10 12:32:02.085434 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:02.085295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tqkdr" event={"ID":"7a8ce703-8992-44dc-a56c-1313710a627d","Type":"ContainerStarted","Data":"1957dcccf4cb92ecb3ae12cd530b7a35e2bd37866fd97f5e9221fe0e5032cea1"}
Sep 10 12:32:03.091522 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:03.091447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tqkdr" event={"ID":"7a8ce703-8992-44dc-a56c-1313710a627d","Type":"ContainerStarted","Data":"7293ff667ec767b92543c0a4c58a41cfa55df651ceb6f35cbd9086ae4852aa3e"}
Sep 10 12:32:04.301821 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:04.301787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:32:04.303985 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:04.303967    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1bf1ff93-c28e-404e-b894-b82f73484d09-metrics-tls\") pod \"dns-default-5qwhw\" (UID: \"1bf1ff93-c28e-404e-b894-b82f73484d09\") " pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:32:04.576419 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:04.576319    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sgmkk\""
Sep 10 12:32:04.583709 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:04.583687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:32:04.859323 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:04.859237    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5qwhw"]
Sep 10 12:32:04.863371 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:32:04.863343    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf1ff93_c28e_404e_b894_b82f73484d09.slice/crio-1d00f6bdd2be7291cd424efb9728a738a500095733996366d4e68b53fbbade54 WatchSource:0}: Error finding container 1d00f6bdd2be7291cd424efb9728a738a500095733996366d4e68b53fbbade54: Status 404 returned error can't find the container with id 1d00f6bdd2be7291cd424efb9728a738a500095733996366d4e68b53fbbade54
Sep 10 12:32:05.099179 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:05.099138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5qwhw" event={"ID":"1bf1ff93-c28e-404e-b894-b82f73484d09","Type":"ContainerStarted","Data":"1d00f6bdd2be7291cd424efb9728a738a500095733996366d4e68b53fbbade54"}
Sep 10 12:32:05.101127 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:05.101105    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tqkdr" event={"ID":"7a8ce703-8992-44dc-a56c-1313710a627d","Type":"ContainerStarted","Data":"9d50796f5e601a6dcd9aab69b4e4339e216350e4491ece00fe4be1c203770198"}
Sep 10 12:32:07.108869 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:07.108836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5qwhw" event={"ID":"1bf1ff93-c28e-404e-b894-b82f73484d09","Type":"ContainerStarted","Data":"28138f0b6535912fa87f0adaf257ccd10daaf3b07b3f26f1ced0e767c9366b92"}
Sep 10 12:32:07.109253 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:07.108886    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5qwhw" event={"ID":"1bf1ff93-c28e-404e-b894-b82f73484d09","Type":"ContainerStarted","Data":"1b78a645e1f312458c6aa0a6f2c904c7f9e9dc1ebca39224447ec506b1a147ef"}
Sep 10 12:32:07.109253 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:07.108930    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:32:07.133586 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:07.133515    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5qwhw" podStartSLOduration=65.63168395 podStartE2EDuration="1m7.133501316s" totalImagesPullingTime="1.501817366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:07.132723614 +0000 UTC m=+101.057635102" watchObservedRunningTime="2026-09-10 12:32:07.133501316 +0000 UTC m=+101.058412804"
Sep 10 12:32:07.133881 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:07.133859    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tqkdr" podStartSLOduration=19.876909469 podStartE2EDuration="22.133851642s" totalImagesPullingTime="2.256942173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:31:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:05.124411051 +0000 UTC m=+99.049322539" watchObservedRunningTime="2026-09-10 12:32:07.133851642 +0000 UTC m=+101.058763130"
Sep 10 12:32:09.006384 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:09.006354    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jz9tr"
Sep 10 12:32:10.256154 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.256120    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-ktd7n"]
Sep 10 12:32:10.290727 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.290698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-ktd7n"
Sep 10 12:32:10.292096 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.291947    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"]
Sep 10 12:32:10.296877 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.296849    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 12:32:10.297850 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.297827    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 12:32:10.298466 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.298445    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-mlztg\""
Sep 10 12:32:10.306532 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.306511    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-ktd7n"]
Sep 10 12:32:10.306644 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.306629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"
Sep 10 12:32:10.330572 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.330265    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 12:32:10.330572 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.330543    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 12:32:10.330828 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.330765    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 12:32:10.333839 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.330944    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 12:32:10.333839 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.331401    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"]
Sep 10 12:32:10.408341 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.408308    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58b575c474-nfrb2"]
Sep 10 12:32:10.448796 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.448762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbeb5cb6-7558-444a-b370-6346e497fea4-tmp\") pod \"klusterlet-addon-workmgr-799c5c98b6-r9npd\" (UID: \"bbeb5cb6-7558-444a-b370-6346e497fea4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"
Sep 10 12:32:10.448975 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.448828    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9-nginx-conf\") pod \"networking-console-plugin-79977f9844-ktd7n\" (UID: \"a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-ktd7n"
Sep 10 12:32:10.448975 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.448881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-ktd7n\" (UID: \"a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-ktd7n"
Sep 10 12:32:10.448975 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.448939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bbeb5cb6-7558-444a-b370-6346e497fea4-klusterlet-config\") pod \"klusterlet-addon-workmgr-799c5c98b6-r9npd\" (UID: \"bbeb5cb6-7558-444a-b370-6346e497fea4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"
Sep 10 12:32:10.449134 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.448989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnpfd\" (UniqueName: \"kubernetes.io/projected/bbeb5cb6-7558-444a-b370-6346e497fea4-kube-api-access-cnpfd\") pod \"klusterlet-addon-workmgr-799c5c98b6-r9npd\" (UID: \"bbeb5cb6-7558-444a-b370-6346e497fea4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"
Sep 10 12:32:10.549812 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.549731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbeb5cb6-7558-444a-b370-6346e497fea4-tmp\") pod \"klusterlet-addon-workmgr-799c5c98b6-r9npd\" (UID: \"bbeb5cb6-7558-444a-b370-6346e497fea4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"
Sep 10 12:32:10.549812 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.549787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9-nginx-conf\") pod \"networking-console-plugin-79977f9844-ktd7n\" (UID: \"a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-ktd7n"
Sep 10 12:32:10.550015 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.549813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-ktd7n\" (UID: \"a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-ktd7n"
Sep 10 12:32:10.550015 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.549842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bbeb5cb6-7558-444a-b370-6346e497fea4-klusterlet-config\") pod \"klusterlet-addon-workmgr-799c5c98b6-r9npd\" (UID: \"bbeb5cb6-7558-444a-b370-6346e497fea4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"
Sep 10 12:32:10.550015 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.549876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnpfd\" (UniqueName: \"kubernetes.io/projected/bbeb5cb6-7558-444a-b370-6346e497fea4-kube-api-access-cnpfd\") pod \"klusterlet-addon-workmgr-799c5c98b6-r9npd\" (UID: \"bbeb5cb6-7558-444a-b370-6346e497fea4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"
Sep 10 12:32:10.550159 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.550140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bbeb5cb6-7558-444a-b370-6346e497fea4-tmp\") pod \"klusterlet-addon-workmgr-799c5c98b6-r9npd\" (UID: \"bbeb5cb6-7558-444a-b370-6346e497fea4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"
Sep 10 12:32:10.550575 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.550519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9-nginx-conf\") pod \"networking-console-plugin-79977f9844-ktd7n\" (UID: \"a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-ktd7n"
Sep 10 12:32:10.552333 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.552306    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-ktd7n\" (UID: \"a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9\") " pod="openshift-network-console/networking-console-plugin-79977f9844-ktd7n"
Sep 10 12:32:10.552496 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.552476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bbeb5cb6-7558-444a-b370-6346e497fea4-klusterlet-config\") pod \"klusterlet-addon-workmgr-799c5c98b6-r9npd\" (UID: \"bbeb5cb6-7558-444a-b370-6346e497fea4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"
Sep 10 12:32:10.557875 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.557850    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64fb4d4586-kbrd6"]
Sep 10 12:32:10.584448 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.584423    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.596884 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.596859    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64fb4d4586-kbrd6"]
Sep 10 12:32:10.599250 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.599223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnpfd\" (UniqueName: \"kubernetes.io/projected/bbeb5cb6-7558-444a-b370-6346e497fea4-kube-api-access-cnpfd\") pod \"klusterlet-addon-workmgr-799c5c98b6-r9npd\" (UID: \"bbeb5cb6-7558-444a-b370-6346e497fea4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"
Sep 10 12:32:10.601024 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.601004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-ktd7n"
Sep 10 12:32:10.630684 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.630662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"
Sep 10 12:32:10.751328 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.751298    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/614191f8-6294-44de-b18f-37d7f71e8e0d-registry-tls\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.751485 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.751350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/614191f8-6294-44de-b18f-37d7f71e8e0d-bound-sa-token\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.751485 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.751380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/614191f8-6294-44de-b18f-37d7f71e8e0d-image-registry-private-configuration\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.751485 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.751399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/614191f8-6294-44de-b18f-37d7f71e8e0d-ca-trust-extracted\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.751485 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.751414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/614191f8-6294-44de-b18f-37d7f71e8e0d-trusted-ca\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.751485 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.751443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqdk8\" (UniqueName: \"kubernetes.io/projected/614191f8-6294-44de-b18f-37d7f71e8e0d-kube-api-access-fqdk8\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.751665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.751533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/614191f8-6294-44de-b18f-37d7f71e8e0d-registry-certificates\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.751665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.751585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/614191f8-6294-44de-b18f-37d7f71e8e0d-installation-pull-secrets\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.852219 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.852135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/614191f8-6294-44de-b18f-37d7f71e8e0d-registry-certificates\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.852219 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.852171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/614191f8-6294-44de-b18f-37d7f71e8e0d-installation-pull-secrets\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.852219 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.852196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/614191f8-6294-44de-b18f-37d7f71e8e0d-registry-tls\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.852475 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.852233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/614191f8-6294-44de-b18f-37d7f71e8e0d-bound-sa-token\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.852475 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.852266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/614191f8-6294-44de-b18f-37d7f71e8e0d-image-registry-private-configuration\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.852475 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.852285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/614191f8-6294-44de-b18f-37d7f71e8e0d-ca-trust-extracted\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.852475 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.852302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/614191f8-6294-44de-b18f-37d7f71e8e0d-trusted-ca\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.852475 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.852339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqdk8\" (UniqueName: \"kubernetes.io/projected/614191f8-6294-44de-b18f-37d7f71e8e0d-kube-api-access-fqdk8\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.852857 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.852832    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/614191f8-6294-44de-b18f-37d7f71e8e0d-ca-trust-extracted\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.853272 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.853248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/614191f8-6294-44de-b18f-37d7f71e8e0d-trusted-ca\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.853366 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.853248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/614191f8-6294-44de-b18f-37d7f71e8e0d-registry-certificates\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.854704 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.854678    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/614191f8-6294-44de-b18f-37d7f71e8e0d-registry-tls\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.854704 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.854696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/614191f8-6294-44de-b18f-37d7f71e8e0d-installation-pull-secrets\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.854827 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.854736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/614191f8-6294-44de-b18f-37d7f71e8e0d-image-registry-private-configuration\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.869492 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.869467    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/614191f8-6294-44de-b18f-37d7f71e8e0d-bound-sa-token\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.874991 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.874962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqdk8\" (UniqueName: \"kubernetes.io/projected/614191f8-6294-44de-b18f-37d7f71e8e0d-kube-api-access-fqdk8\") pod \"image-registry-64fb4d4586-kbrd6\" (UID: \"614191f8-6294-44de-b18f-37d7f71e8e0d\") " pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.893101 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.893077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:10.925732 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.925691    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-ktd7n"]
Sep 10 12:32:10.930472 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:32:10.930437    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cf6d4b_edf0_48a6_8bfa_26aab374b6f9.slice/crio-7f106376e631ffad5dfaf6456096828e8c667836ca1251c9889c33715753095f WatchSource:0}: Error finding container 7f106376e631ffad5dfaf6456096828e8c667836ca1251c9889c33715753095f: Status 404 returned error can't find the container with id 7f106376e631ffad5dfaf6456096828e8c667836ca1251c9889c33715753095f
Sep 10 12:32:10.942724 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:10.942696    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"]
Sep 10 12:32:10.947150 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:32:10.947125    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbeb5cb6_7558_444a_b370_6346e497fea4.slice/crio-52f58e12e235f34900472074e00928e2fb31d63672052b6276f657651bbadcfa WatchSource:0}: Error finding container 52f58e12e235f34900472074e00928e2fb31d63672052b6276f657651bbadcfa: Status 404 returned error can't find the container with id 52f58e12e235f34900472074e00928e2fb31d63672052b6276f657651bbadcfa
Sep 10 12:32:11.124085 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:11.123990    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd" event={"ID":"bbeb5cb6-7558-444a-b370-6346e497fea4","Type":"ContainerStarted","Data":"52f58e12e235f34900472074e00928e2fb31d63672052b6276f657651bbadcfa"}
Sep 10 12:32:11.125113 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:11.125082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-ktd7n" event={"ID":"a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9","Type":"ContainerStarted","Data":"7f106376e631ffad5dfaf6456096828e8c667836ca1251c9889c33715753095f"}
Sep 10 12:32:11.285430 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:11.285403    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64fb4d4586-kbrd6"]
Sep 10 12:32:12.129867 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:12.129826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6" event={"ID":"614191f8-6294-44de-b18f-37d7f71e8e0d","Type":"ContainerStarted","Data":"85bee411a7df38184f5e120142d0c664616ccb001f18c0d9cab471d1b8fd500b"}
Sep 10 12:32:12.130073 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:12.129888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6" event={"ID":"614191f8-6294-44de-b18f-37d7f71e8e0d","Type":"ContainerStarted","Data":"f5299079e9f4373e65ce832bc7055c098a6d3b2cadf2df428a63006d1eadeedf"}
Sep 10 12:32:12.130073 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:12.129932    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:13.134224 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:13.134184    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-ktd7n" event={"ID":"a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9","Type":"ContainerStarted","Data":"247741e165e2062fcf0be0e6682bf1725425893efb85020a13a57920e2a19409"}
Sep 10 12:32:13.177013 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:13.176962    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6" podStartSLOduration=3.176944678 podStartE2EDuration="3.176944678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:12.171275803 +0000 UTC m=+106.096187493" watchObservedRunningTime="2026-09-10 12:32:13.176944678 +0000 UTC m=+107.101856165"
Sep 10 12:32:13.272080 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:13.272037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:32:13.274510 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:13.274486    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b897d389-1590-4c48-9504-24c834416b86-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-nbc4r\" (UID: \"b897d389-1590-4c48-9504-24c834416b86\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:32:13.563493 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:13.563459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"
Sep 10 12:32:13.852149 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:13.852055    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-ktd7n" podStartSLOduration=2.553471359 podStartE2EDuration="3.852036301s" totalImagesPullingTime="1.298564942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:13.176877201 +0000 UTC m=+107.101788691" watchObservedRunningTime="2026-09-10 12:32:13.852036301 +0000 UTC m=+107.776947786"
Sep 10 12:32:13.852512 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:13.852494    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r"]
Sep 10 12:32:13.854903 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:32:13.854870    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb897d389_1590_4c48_9504_24c834416b86.slice/crio-dca7082a1c1f1fd674e06552f50ff99b95b044b934954d54942e67f63f6478a7 WatchSource:0}: Error finding container dca7082a1c1f1fd674e06552f50ff99b95b044b934954d54942e67f63f6478a7: Status 404 returned error can't find the container with id dca7082a1c1f1fd674e06552f50ff99b95b044b934954d54942e67f63f6478a7
Sep 10 12:32:14.139242 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:14.139164    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r" event={"ID":"b897d389-1590-4c48-9504-24c834416b86","Type":"ContainerStarted","Data":"dca7082a1c1f1fd674e06552f50ff99b95b044b934954d54942e67f63f6478a7"}
Sep 10 12:32:16.148590 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:16.148536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r" event={"ID":"b897d389-1590-4c48-9504-24c834416b86","Type":"ContainerStarted","Data":"4bf028af2bb48a4b28dc0a0aaf33b51a2efd4075c9b7795721150f418f7856fd"}
Sep 10 12:32:16.183426 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:16.183370    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-nbc4r" podStartSLOduration=33.689210266 podStartE2EDuration="35.183355349s" totalImagesPullingTime="1.494145083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:16.182745089 +0000 UTC m=+110.107656579" watchObservedRunningTime="2026-09-10 12:32:16.183355349 +0000 UTC m=+110.108266838"
Sep 10 12:32:17.115688 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:17.115652    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5qwhw"
Sep 10 12:32:20.107585 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.107531    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-svtzz"]
Sep 10 12:32:20.110129 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.110104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:20.112490 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.112460    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 12:32:20.112620 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.112467    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 12:32:20.114138 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.114115    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9v789\""
Sep 10 12:32:20.114262 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.114237    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 12:32:20.122192 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.122166    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-svtzz"]
Sep 10 12:32:20.227426 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.227386    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj5lm\" (UniqueName: \"kubernetes.io/projected/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-kube-api-access-tj5lm\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:20.227654 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.227459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:20.227654 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.227521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:20.227654 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.227547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:20.328924 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.328879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:20.329124 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.328932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:20.329124 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.328999    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj5lm\" (UniqueName: \"kubernetes.io/projected/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-kube-api-access-tj5lm\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:20.329124 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:32:20.329021    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 10 12:32:20.329124 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.329048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:20.329124 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:32:20.329114    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-prometheus-operator-tls podName:a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9 nodeName:}" failed. No retries permitted until 2026-09-10 12:32:20.829092087 +0000 UTC m=+114.754003571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-svtzz" (UID: "a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9") : secret "prometheus-operator-tls" not found
Sep 10 12:32:20.329836 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.329791    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:20.331909 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.331890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:20.338437 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.338410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj5lm\" (UniqueName: \"kubernetes.io/projected/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-kube-api-access-tj5lm\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:20.414750 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.414677    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:32:20.834364 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:20.834319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:20.834580 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:32:20.834452    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 10 12:32:20.834580 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:32:20.834536    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-prometheus-operator-tls podName:a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9 nodeName:}" failed. No retries permitted until 2026-09-10 12:32:21.834517147 +0000 UTC m=+115.759428635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-svtzz" (UID: "a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9") : secret "prometheus-operator-tls" not found
Sep 10 12:32:21.168791 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:21.168758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd" event={"ID":"bbeb5cb6-7558-444a-b370-6346e497fea4","Type":"ContainerStarted","Data":"c1c2d6aa6b3385d395d738d62a896aa5a92712fa3f6d1cb08fb642eb78860855"}
Sep 10 12:32:21.169322 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:21.169300    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"
Sep 10 12:32:21.171096 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:21.171068    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd"
Sep 10 12:32:21.189009 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:21.188965    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-799c5c98b6-r9npd" podStartSLOduration=1.214722882 podStartE2EDuration="11.188951558s" totalImagesPullingTime="9.974228676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:21.188063464 +0000 UTC m=+115.112974954" watchObservedRunningTime="2026-09-10 12:32:21.188951558 +0000 UTC m=+115.113863037"
Sep 10 12:32:21.844179 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:21.844136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:21.846470 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:21.846449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-svtzz\" (UID: \"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:21.922712 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:21.922663    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz"
Sep 10 12:32:22.192855 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:22.192824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-svtzz"]
Sep 10 12:32:22.195585 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:32:22.195543    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f9d19a_ba01_4f53_9e2f_ba110c9b6fc9.slice/crio-dab34132c6821aaef4c2ff6c379b6856b98ea5bb0ac428a7514bee74c48289ec WatchSource:0}: Error finding container dab34132c6821aaef4c2ff6c379b6856b98ea5bb0ac428a7514bee74c48289ec: Status 404 returned error can't find the container with id dab34132c6821aaef4c2ff6c379b6856b98ea5bb0ac428a7514bee74c48289ec
Sep 10 12:32:23.177089 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:23.177041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz" event={"ID":"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9","Type":"ContainerStarted","Data":"dab34132c6821aaef4c2ff6c379b6856b98ea5bb0ac428a7514bee74c48289ec"}
Sep 10 12:32:24.182577 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:24.182515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz" event={"ID":"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9","Type":"ContainerStarted","Data":"6f7e8cc26d74c3a3551dc14549233df4d1667426dd2b59e5e4df11320736c10b"}
Sep 10 12:32:24.182577 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:24.182581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz" event={"ID":"a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9","Type":"ContainerStarted","Data":"8b76f2ce5bb3ecaf484fbdfbf652ed046dd45f5762323db9d221719cd7c052d2"}
Sep 10 12:32:24.202946 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:24.202890    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-svtzz" podStartSLOduration=2.8752546949999997 podStartE2EDuration="4.202869507s" totalImagesPullingTime="1.327614812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:24.202160305 +0000 UTC m=+118.127071794" watchObservedRunningTime="2026-09-10 12:32:24.202869507 +0000 UTC m=+118.127780994"
Sep 10 12:32:26.574814 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.574778    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-77whj"]
Sep 10 12:32:26.578456 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.578431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.581280 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.581261    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 12:32:26.581647 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.581627    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 12:32:26.581744 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.581666    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dxq69\""
Sep 10 12:32:26.581862 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.581848    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 12:32:26.587622 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.587600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-tls\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.587720 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.587641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7nbs\" (UniqueName: \"kubernetes.io/projected/1a139624-4246-47bd-acb1-d8acd9c5c272-kube-api-access-n7nbs\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.587720 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.587671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a139624-4246-47bd-acb1-d8acd9c5c272-metrics-client-ca\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.587720 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.587699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-wtmp\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.587879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.587744    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-accelerators-collector-config\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.587879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.587777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a139624-4246-47bd-acb1-d8acd9c5c272-root\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.587879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.587801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.587879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.587835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-textfile\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.588104 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.587921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a139624-4246-47bd-acb1-d8acd9c5c272-sys\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.688867 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.688827    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a139624-4246-47bd-acb1-d8acd9c5c272-sys\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689043 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.688902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-tls\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689043 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.688935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7nbs\" (UniqueName: \"kubernetes.io/projected/1a139624-4246-47bd-acb1-d8acd9c5c272-kube-api-access-n7nbs\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689043 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.688938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a139624-4246-47bd-acb1-d8acd9c5c272-sys\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689043 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.688970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a139624-4246-47bd-acb1-d8acd9c5c272-metrics-client-ca\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689043 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:32:26.689024    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 12:32:26.689308 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.689022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-wtmp\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689308 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:32:26.689089    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-tls podName:1a139624-4246-47bd-acb1-d8acd9c5c272 nodeName:}" failed. No retries permitted until 2026-09-10 12:32:27.189065915 +0000 UTC m=+121.113977396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-tls") pod "node-exporter-77whj" (UID: "1a139624-4246-47bd-acb1-d8acd9c5c272") : secret "node-exporter-tls" not found
Sep 10 12:32:26.689308 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.689117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-accelerators-collector-config\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689308 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.689144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-wtmp\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689308 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.689153    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a139624-4246-47bd-acb1-d8acd9c5c272-root\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689308 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.689176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689308 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.689219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-textfile\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689308 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.689297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1a139624-4246-47bd-acb1-d8acd9c5c272-root\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689826 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.689609    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a139624-4246-47bd-acb1-d8acd9c5c272-metrics-client-ca\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689826 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.689669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-textfile\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.689826 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.689721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-accelerators-collector-config\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.691639 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.691613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:26.713812 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:26.713785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7nbs\" (UniqueName: \"kubernetes.io/projected/1a139624-4246-47bd-acb1-d8acd9c5c272-kube-api-access-n7nbs\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:27.194470 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:27.194433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-tls\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:27.196862 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:27.196838    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1a139624-4246-47bd-acb1-d8acd9c5c272-node-exporter-tls\") pod \"node-exporter-77whj\" (UID: \"1a139624-4246-47bd-acb1-d8acd9c5c272\") " pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:27.488844 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:27.488756    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-77whj"
Sep 10 12:32:27.500672 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:32:27.500639    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a139624_4246_47bd_acb1_d8acd9c5c272.slice/crio-6012d1965f44a9f7346a10c979f828ed1b941806840e39e39adb03d66f9b3348 WatchSource:0}: Error finding container 6012d1965f44a9f7346a10c979f828ed1b941806840e39e39adb03d66f9b3348: Status 404 returned error can't find the container with id 6012d1965f44a9f7346a10c979f828ed1b941806840e39e39adb03d66f9b3348
Sep 10 12:32:28.197964 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:28.197921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77whj" event={"ID":"1a139624-4246-47bd-acb1-d8acd9c5c272","Type":"ContainerStarted","Data":"6012d1965f44a9f7346a10c979f828ed1b941806840e39e39adb03d66f9b3348"}
Sep 10 12:32:29.203104 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.203065    2563 generic.go:356] "Generic (PLEG): container finished" podID="1a139624-4246-47bd-acb1-d8acd9c5c272" containerID="b19dfeda77df9bb4735db89768509dc39932f3fd22f51287d0316a782b99aec6" exitCode=0
Sep 10 12:32:29.203468 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.203156    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77whj" event={"ID":"1a139624-4246-47bd-acb1-d8acd9c5c272","Type":"ContainerDied","Data":"b19dfeda77df9bb4735db89768509dc39932f3fd22f51287d0316a782b99aec6"}
Sep 10 12:32:29.583952 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.583863    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7cd78b988c-snz6p"]
Sep 10 12:32:29.587373 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.587354    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.590292 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.590265    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-cg7tn\""
Sep 10 12:32:29.590408 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.590308    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 10 12:32:29.590408 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.590265    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 10 12:32:29.590408 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.590377    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 10 12:32:29.590408 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.590387    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f9oh0uvruecig\""
Sep 10 12:32:29.590614 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.590311    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 10 12:32:29.590614 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.590317    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 10 12:32:29.603843 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.603815    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7cd78b988c-snz6p"]
Sep 10 12:32:29.612885 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.612855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.613002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.612913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.613002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.612939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-metrics-client-ca\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.613002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.612977    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d57dl\" (UniqueName: \"kubernetes.io/projected/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-kube-api-access-d57dl\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.613161 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.613035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.613161 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.613066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-grpc-tls\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.613161 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.613118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-tls\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.613302 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.613164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.714103 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.714066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.714294 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.714113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-metrics-client-ca\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.714294 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.714146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d57dl\" (UniqueName: \"kubernetes.io/projected/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-kube-api-access-d57dl\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.714294 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.714219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.714294 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.714281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-grpc-tls\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.714505 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.714339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-tls\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.714505 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.714380    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.714505 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.714475    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.715041 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.715016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-metrics-client-ca\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.717086 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.717046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.717700 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.717169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.717700 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.717595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-grpc-tls\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.717700 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.717659    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.717877 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.717740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.718991 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.718973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-secret-thanos-querier-tls\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.724296 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.724267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d57dl\" (UniqueName: \"kubernetes.io/projected/41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc-kube-api-access-d57dl\") pod \"thanos-querier-7cd78b988c-snz6p\" (UID: \"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc\") " pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:29.899126 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:29.899027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:30.177772 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:30.177746    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7cd78b988c-snz6p"]
Sep 10 12:32:30.180190 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:32:30.180162    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c0d533_a07e_49bc_a7a5_8ea2e97e3dcc.slice/crio-c14205ad205d2fabad137c0577da27735960bf2e64f5d8b5403d2fda4e1c4887 WatchSource:0}: Error finding container c14205ad205d2fabad137c0577da27735960bf2e64f5d8b5403d2fda4e1c4887: Status 404 returned error can't find the container with id c14205ad205d2fabad137c0577da27735960bf2e64f5d8b5403d2fda4e1c4887
Sep 10 12:32:30.209084 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:30.209050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77whj" event={"ID":"1a139624-4246-47bd-acb1-d8acd9c5c272","Type":"ContainerStarted","Data":"9ef9e01268260b3be717db3f727190c911dbdee19c2fa77e3a61288d5907f23a"}
Sep 10 12:32:30.209496 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:30.209112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-77whj" event={"ID":"1a139624-4246-47bd-acb1-d8acd9c5c272","Type":"ContainerStarted","Data":"14fc9b4ee194d3cd9be82553c8df3ff24a831dbcaae72b9f7789a7f8f56b5a8c"}
Sep 10 12:32:30.210201 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:30.210181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p" event={"ID":"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc","Type":"ContainerStarted","Data":"c14205ad205d2fabad137c0577da27735960bf2e64f5d8b5403d2fda4e1c4887"}
Sep 10 12:32:30.231211 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:30.231159    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-77whj" podStartSLOduration=3.14667051 podStartE2EDuration="4.231143717s" totalImagesPullingTime="1.023846116s" totalInitContainerRuntime="60.627091ms" isStatefulPod=true podCreationTimestamp="2026-09-10 12:32:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:30.229538994 +0000 UTC m=+124.154450482" watchObservedRunningTime="2026-09-10 12:32:30.231143717 +0000 UTC m=+124.156055204"
Sep 10 12:32:31.240467 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:31.240421    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p"]
Sep 10 12:32:31.244034 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:31.244008    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p"
Sep 10 12:32:31.252299 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:31.251858    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-h5kfc\""
Sep 10 12:32:31.252699 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:31.252507    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 10 12:32:31.254888 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:31.254858    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p"]
Sep 10 12:32:31.329127 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:31.329069    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrtqs\" (UniqueName: \"kubernetes.io/projected/30ea156b-7223-4fd1-9b04-2ffde135f99c-kube-api-access-vrtqs\") pod \"monitoring-plugin-85bbbbbf4-cg64p\" (UID: \"30ea156b-7223-4fd1-9b04-2ffde135f99c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p"
Sep 10 12:32:31.329307 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:31.329199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30ea156b-7223-4fd1-9b04-2ffde135f99c-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-cg64p\" (UID: \"30ea156b-7223-4fd1-9b04-2ffde135f99c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p"
Sep 10 12:32:31.430346 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:31.430305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrtqs\" (UniqueName: \"kubernetes.io/projected/30ea156b-7223-4fd1-9b04-2ffde135f99c-kube-api-access-vrtqs\") pod \"monitoring-plugin-85bbbbbf4-cg64p\" (UID: \"30ea156b-7223-4fd1-9b04-2ffde135f99c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p"
Sep 10 12:32:31.430489 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:31.430369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30ea156b-7223-4fd1-9b04-2ffde135f99c-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-cg64p\" (UID: \"30ea156b-7223-4fd1-9b04-2ffde135f99c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p"
Sep 10 12:32:31.432868 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:31.432844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/30ea156b-7223-4fd1-9b04-2ffde135f99c-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-cg64p\" (UID: \"30ea156b-7223-4fd1-9b04-2ffde135f99c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p"
Sep 10 12:32:31.441041 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:31.441016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrtqs\" (UniqueName: \"kubernetes.io/projected/30ea156b-7223-4fd1-9b04-2ffde135f99c-kube-api-access-vrtqs\") pod \"monitoring-plugin-85bbbbbf4-cg64p\" (UID: \"30ea156b-7223-4fd1-9b04-2ffde135f99c\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p"
Sep 10 12:32:31.562394 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:31.562304    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p"
Sep 10 12:32:31.840798 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:31.840724    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p"]
Sep 10 12:32:32.133392 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:32:32.133329    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ea156b_7223_4fd1_9b04_2ffde135f99c.slice/crio-700a61a789f12a518d6def421cb46fbe9ae6586ee80bdf3202a224d1a802e3ac WatchSource:0}: Error finding container 700a61a789f12a518d6def421cb46fbe9ae6586ee80bdf3202a224d1a802e3ac: Status 404 returned error can't find the container with id 700a61a789f12a518d6def421cb46fbe9ae6586ee80bdf3202a224d1a802e3ac
Sep 10 12:32:32.219608 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:32.219579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p" event={"ID":"30ea156b-7223-4fd1-9b04-2ffde135f99c","Type":"ContainerStarted","Data":"700a61a789f12a518d6def421cb46fbe9ae6586ee80bdf3202a224d1a802e3ac"}
Sep 10 12:32:33.139840 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:33.139803    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64fb4d4586-kbrd6"
Sep 10 12:32:33.227109 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:33.227057    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p" event={"ID":"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc","Type":"ContainerStarted","Data":"f3bebae9a6b2095fcfeb694ef71aec8ee0b383e0beb565294650a82f73af3e15"}
Sep 10 12:32:33.227281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:33.227138    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p" event={"ID":"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc","Type":"ContainerStarted","Data":"3ea57ff0f46eed750f6cec3ade91b2404f20ff93d8c84fe3ef74622f638130bd"}
Sep 10 12:32:33.227281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:33.227175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p" event={"ID":"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc","Type":"ContainerStarted","Data":"77117f1fa64b0d21fc627ed8e4abd3745da3d714740e2652803a63353ceca425"}
Sep 10 12:32:34.232030 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:34.231983    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p" event={"ID":"30ea156b-7223-4fd1-9b04-2ffde135f99c","Type":"ContainerStarted","Data":"a5e9d1304424dd311050ad8a0eb647061296eb85ae5877c1afe4010afe442ebc"}
Sep 10 12:32:34.232448 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:34.232163    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p"
Sep 10 12:32:34.235300 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:34.235275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p" event={"ID":"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc","Type":"ContainerStarted","Data":"350417a1e6ddba899efecb504d2c9ce7dbdbb3d7ba13a3348ae65e0de839f455"}
Sep 10 12:32:34.235410 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:34.235340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p" event={"ID":"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc","Type":"ContainerStarted","Data":"ebe4d204e3df15f25984328dd32ee07aab12000ee3ce7b5e9a14d70512782946"}
Sep 10 12:32:34.235410 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:34.235381    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p" event={"ID":"41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc","Type":"ContainerStarted","Data":"a34a417d67c84ed69ec9debdc08026f175c87f02e07b50bd0a0ea8d2a88cdf59"}
Sep 10 12:32:34.235477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:34.235425    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:34.237214 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:34.237194    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p"
Sep 10 12:32:34.254809 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:34.254765    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-cg64p" podStartSLOduration=1.7822070700000001 podStartE2EDuration="3.254750434s" totalImagesPullingTime="1.472543364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:34.251456651 +0000 UTC m=+128.176368150" watchObservedRunningTime="2026-09-10 12:32:34.254750434 +0000 UTC m=+128.179661953"
Sep 10 12:32:34.283402 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:34.283342    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p" podStartSLOduration=2.044595445 podStartE2EDuration="5.283323081s" totalImagesPullingTime="3.238727636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:34.282166056 +0000 UTC m=+128.207077555" watchObservedRunningTime="2026-09-10 12:32:34.283323081 +0000 UTC m=+128.208234570"
Sep 10 12:32:35.427182 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.427111    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-58b575c474-nfrb2" podUID="2456a6c7-1269-4dca-a837-1619420c6fd4" containerName="registry" containerID="cri-o://509cbf0eb8c34455a949215a2addc5fac93f41c46f20460ad7236a19575c1979" gracePeriod=30
Sep 10 12:32:35.661126 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.661102    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:32:35.774007 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.773920    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2456a6c7-1269-4dca-a837-1619420c6fd4-ca-trust-extracted\") pod \"2456a6c7-1269-4dca-a837-1619420c6fd4\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") "
Sep 10 12:32:35.774007 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.773961    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-bound-sa-token\") pod \"2456a6c7-1269-4dca-a837-1619420c6fd4\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") "
Sep 10 12:32:35.774007 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.774002    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2456a6c7-1269-4dca-a837-1619420c6fd4-image-registry-private-configuration\") pod \"2456a6c7-1269-4dca-a837-1619420c6fd4\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") "
Sep 10 12:32:35.774281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.774019    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2456a6c7-1269-4dca-a837-1619420c6fd4-installation-pull-secrets\") pod \"2456a6c7-1269-4dca-a837-1619420c6fd4\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") "
Sep 10 12:32:35.774281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.774055    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sk6h\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-kube-api-access-8sk6h\") pod \"2456a6c7-1269-4dca-a837-1619420c6fd4\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") "
Sep 10 12:32:35.774281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.774082    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls\") pod \"2456a6c7-1269-4dca-a837-1619420c6fd4\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") "
Sep 10 12:32:35.774281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.774170    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-certificates\") pod \"2456a6c7-1269-4dca-a837-1619420c6fd4\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") "
Sep 10 12:32:35.774281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.774212    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2456a6c7-1269-4dca-a837-1619420c6fd4-trusted-ca\") pod \"2456a6c7-1269-4dca-a837-1619420c6fd4\" (UID: \"2456a6c7-1269-4dca-a837-1619420c6fd4\") "
Sep 10 12:32:35.774929 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.774863    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2456a6c7-1269-4dca-a837-1619420c6fd4" (UID: "2456a6c7-1269-4dca-a837-1619420c6fd4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:32:35.774929 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.774880    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2456a6c7-1269-4dca-a837-1619420c6fd4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2456a6c7-1269-4dca-a837-1619420c6fd4" (UID: "2456a6c7-1269-4dca-a837-1619420c6fd4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:32:35.777048 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.777013    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2456a6c7-1269-4dca-a837-1619420c6fd4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2456a6c7-1269-4dca-a837-1619420c6fd4" (UID: "2456a6c7-1269-4dca-a837-1619420c6fd4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:32:35.777147 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.777018    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2456a6c7-1269-4dca-a837-1619420c6fd4" (UID: "2456a6c7-1269-4dca-a837-1619420c6fd4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:32:35.777147 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.777084    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2456a6c7-1269-4dca-a837-1619420c6fd4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2456a6c7-1269-4dca-a837-1619420c6fd4" (UID: "2456a6c7-1269-4dca-a837-1619420c6fd4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:32:35.777147 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.777092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-kube-api-access-8sk6h" (OuterVolumeSpecName: "kube-api-access-8sk6h") pod "2456a6c7-1269-4dca-a837-1619420c6fd4" (UID: "2456a6c7-1269-4dca-a837-1619420c6fd4"). InnerVolumeSpecName "kube-api-access-8sk6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:32:35.777689 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.777657    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2456a6c7-1269-4dca-a837-1619420c6fd4" (UID: "2456a6c7-1269-4dca-a837-1619420c6fd4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:32:35.785401 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.785373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2456a6c7-1269-4dca-a837-1619420c6fd4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2456a6c7-1269-4dca-a837-1619420c6fd4" (UID: "2456a6c7-1269-4dca-a837-1619420c6fd4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:32:35.875404 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.875368    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2456a6c7-1269-4dca-a837-1619420c6fd4-ca-trust-extracted\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:32:35.875404 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.875397    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-bound-sa-token\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:32:35.875404 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.875408    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2456a6c7-1269-4dca-a837-1619420c6fd4-image-registry-private-configuration\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:32:35.875643 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.875418    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2456a6c7-1269-4dca-a837-1619420c6fd4-installation-pull-secrets\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:32:35.875643 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.875429    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sk6h\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-kube-api-access-8sk6h\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:32:35.875643 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.875438    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-tls\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:32:35.875643 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.875447    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2456a6c7-1269-4dca-a837-1619420c6fd4-registry-certificates\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:32:35.875643 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:35.875455    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2456a6c7-1269-4dca-a837-1619420c6fd4-trusted-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:32:36.242853 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.242817    2563 generic.go:356] "Generic (PLEG): container finished" podID="2456a6c7-1269-4dca-a837-1619420c6fd4" containerID="509cbf0eb8c34455a949215a2addc5fac93f41c46f20460ad7236a19575c1979" exitCode=0
Sep 10 12:32:36.243002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.242860    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58b575c474-nfrb2"
Sep 10 12:32:36.243002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.242904    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58b575c474-nfrb2" event={"ID":"2456a6c7-1269-4dca-a837-1619420c6fd4","Type":"ContainerDied","Data":"509cbf0eb8c34455a949215a2addc5fac93f41c46f20460ad7236a19575c1979"}
Sep 10 12:32:36.243002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.242951    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58b575c474-nfrb2" event={"ID":"2456a6c7-1269-4dca-a837-1619420c6fd4","Type":"ContainerDied","Data":"a91621c5d43410465bfbd03dfd5a2dfb0b128255512515f86cf9118ff9e0dfa0"}
Sep 10 12:32:36.243002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.242973    2563 scope.go:117] "RemoveContainer" containerID="509cbf0eb8c34455a949215a2addc5fac93f41c46f20460ad7236a19575c1979"
Sep 10 12:32:36.251275 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.251259    2563 scope.go:117] "RemoveContainer" containerID="509cbf0eb8c34455a949215a2addc5fac93f41c46f20460ad7236a19575c1979"
Sep 10 12:32:36.251518 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:32:36.251496    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509cbf0eb8c34455a949215a2addc5fac93f41c46f20460ad7236a19575c1979\": container with ID starting with 509cbf0eb8c34455a949215a2addc5fac93f41c46f20460ad7236a19575c1979 not found: ID does not exist" containerID="509cbf0eb8c34455a949215a2addc5fac93f41c46f20460ad7236a19575c1979"
Sep 10 12:32:36.251610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.251530    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509cbf0eb8c34455a949215a2addc5fac93f41c46f20460ad7236a19575c1979"} err="failed to get container status \"509cbf0eb8c34455a949215a2addc5fac93f41c46f20460ad7236a19575c1979\": rpc error: code = NotFound desc = could not find container \"509cbf0eb8c34455a949215a2addc5fac93f41c46f20460ad7236a19575c1979\": container with ID starting with 509cbf0eb8c34455a949215a2addc5fac93f41c46f20460ad7236a19575c1979 not found: ID does not exist"
Sep 10 12:32:36.278377 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.278351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:32:36.280593 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.280571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a04476a3-6d83-40ff-879c-c9a5b620ef44-metrics-certs\") pod \"network-metrics-daemon-vj4n4\" (UID: \"a04476a3-6d83-40ff-879c-c9a5b620ef44\") " pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:32:36.295470 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.295446    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58b575c474-nfrb2"]
Sep 10 12:32:36.301850 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.301828    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58b575c474-nfrb2"]
Sep 10 12:32:36.410519 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.410492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sz48d\""
Sep 10 12:32:36.418820 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.418801    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vj4n4"
Sep 10 12:32:36.703368 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:32:36.703336    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda04476a3_6d83_40ff_879c_c9a5b620ef44.slice/crio-16e069f081bdb0571ea9f0b4357b4d6a27c8280bc7fcb2ecf63a8e13eba5bd29 WatchSource:0}: Error finding container 16e069f081bdb0571ea9f0b4357b4d6a27c8280bc7fcb2ecf63a8e13eba5bd29: Status 404 returned error can't find the container with id 16e069f081bdb0571ea9f0b4357b4d6a27c8280bc7fcb2ecf63a8e13eba5bd29
Sep 10 12:32:36.704068 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.704042    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2456a6c7-1269-4dca-a837-1619420c6fd4" path="/var/lib/kubelet/pods/2456a6c7-1269-4dca-a837-1619420c6fd4/volumes"
Sep 10 12:32:36.704353 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:36.704337    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vj4n4"]
Sep 10 12:32:37.249973 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:37.249927    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vj4n4" event={"ID":"a04476a3-6d83-40ff-879c-c9a5b620ef44","Type":"ContainerStarted","Data":"16e069f081bdb0571ea9f0b4357b4d6a27c8280bc7fcb2ecf63a8e13eba5bd29"}
Sep 10 12:32:38.256448 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:38.256397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vj4n4" event={"ID":"a04476a3-6d83-40ff-879c-c9a5b620ef44","Type":"ContainerStarted","Data":"c44ba5a44c200985163a25abe273c0b8470fe54291f4c2330b9f014c6024827f"}
Sep 10 12:32:38.256971 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:38.256473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vj4n4" event={"ID":"a04476a3-6d83-40ff-879c-c9a5b620ef44","Type":"ContainerStarted","Data":"2889efa7ea6435b20f01719c73389fb6c4cbdd3d3bf359e3091805151acc11a9"}
Sep 10 12:32:38.275113 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:38.275061    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vj4n4" podStartSLOduration=131.327247174 podStartE2EDuration="2m12.275045454s" totalImagesPullingTime="947.79828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:38.27445881 +0000 UTC m=+132.199370301" watchObservedRunningTime="2026-09-10 12:32:38.275045454 +0000 UTC m=+132.199956934"
Sep 10 12:32:40.244854 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:40.244826    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7cd78b988c-snz6p"
Sep 10 12:32:55.323908 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:55.323871    2563 generic.go:356] "Generic (PLEG): container finished" podID="49d13c55-3c7d-4df7-8593-ae68b8b31f77" containerID="39c693ba189c4ddf9450021ea2ae4138c60ba454e5c15d174b3e39c0ebda2d4c" exitCode=0
Sep 10 12:32:55.324314 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:55.323945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz" event={"ID":"49d13c55-3c7d-4df7-8593-ae68b8b31f77","Type":"ContainerDied","Data":"39c693ba189c4ddf9450021ea2ae4138c60ba454e5c15d174b3e39c0ebda2d4c"}
Sep 10 12:32:55.324354 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:55.324315    2563 scope.go:117] "RemoveContainer" containerID="39c693ba189c4ddf9450021ea2ae4138c60ba454e5c15d174b3e39c0ebda2d4c"
Sep 10 12:32:56.329565 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:32:56.329526    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-584zz" event={"ID":"49d13c55-3c7d-4df7-8593-ae68b8b31f77","Type":"ContainerStarted","Data":"3e43458d4026a90efd94f3fc041e881040031c3a501e0bb035676e868701b2cf"}
Sep 10 12:33:20.417604 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:20.417547    2563 generic.go:356] "Generic (PLEG): container finished" podID="0de14be1-91b2-4b99-9a58-d73f2f699570" containerID="b2cc915f675e60768c46d6f2d7f35539d987ba7784e28d469dcdd41cdda79de1" exitCode=0
Sep 10 12:33:20.417964 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:20.417621    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb" event={"ID":"0de14be1-91b2-4b99-9a58-d73f2f699570","Type":"ContainerDied","Data":"b2cc915f675e60768c46d6f2d7f35539d987ba7784e28d469dcdd41cdda79de1"}
Sep 10 12:33:20.417964 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:20.417951    2563 scope.go:117] "RemoveContainer" containerID="b2cc915f675e60768c46d6f2d7f35539d987ba7784e28d469dcdd41cdda79de1"
Sep 10 12:33:21.422923 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:21.422888    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fn2vb" event={"ID":"0de14be1-91b2-4b99-9a58-d73f2f699570","Type":"ContainerStarted","Data":"ffc99260a05fa9b3c62fe9aa3c550fe57f706437fe0e3141b302555381198a83"}
Sep 10 12:33:49.145317 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.145281    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:33:49.145762 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.145567    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2456a6c7-1269-4dca-a837-1619420c6fd4" containerName="registry"
Sep 10 12:33:49.145762 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.145579    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2456a6c7-1269-4dca-a837-1619420c6fd4" containerName="registry"
Sep 10 12:33:49.145762 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.145638    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2456a6c7-1269-4dca-a837-1619420c6fd4" containerName="registry"
Sep 10 12:33:49.148063 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.148036    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.151187 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.151153    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 12:33:49.151298 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.151190    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 12:33:49.153561 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.153529    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 12:33:49.153694 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.153674    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 12:33:49.153772 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.153756    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 12:33:49.154739 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.154719    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 12:33:49.154966 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.154940    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 12:33:49.155118 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.155086    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pjmps\""
Sep 10 12:33:49.155203 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.155125    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 12:33:49.159795 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.159778    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 12:33:49.165948 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.165923    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:33:49.242245 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.242204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d773150f-ac5f-4997-8240-ac31df2cb1b6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.242245 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.242249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6zxj\" (UniqueName: \"kubernetes.io/projected/d773150f-ac5f-4997-8240-ac31df2cb1b6-kube-api-access-h6zxj\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.242498 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.242312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d773150f-ac5f-4997-8240-ac31df2cb1b6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.242498 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.242355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d773150f-ac5f-4997-8240-ac31df2cb1b6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.242498 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.242410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.242498 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.242439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d773150f-ac5f-4997-8240-ac31df2cb1b6-config-out\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.242498 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.242481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-config-volume\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.242777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.242504    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.242777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.242537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-web-config\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.242777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.242598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d773150f-ac5f-4997-8240-ac31df2cb1b6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.242777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.242640    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.242777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.242669    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.242777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.242701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344012 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.343975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d773150f-ac5f-4997-8240-ac31df2cb1b6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344012 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d773150f-ac5f-4997-8240-ac31df2cb1b6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344207 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344207 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d773150f-ac5f-4997-8240-ac31df2cb1b6-config-out\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344207 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-config-volume\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344207 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344207 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-web-config\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344207 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d773150f-ac5f-4997-8240-ac31df2cb1b6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344207 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344520 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344520 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344520 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344312    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d773150f-ac5f-4997-8240-ac31df2cb1b6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344520 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6zxj\" (UniqueName: \"kubernetes.io/projected/d773150f-ac5f-4997-8240-ac31df2cb1b6-kube-api-access-h6zxj\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.344902 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d773150f-ac5f-4997-8240-ac31df2cb1b6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.345061 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.344978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d773150f-ac5f-4997-8240-ac31df2cb1b6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.347422 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.347358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d773150f-ac5f-4997-8240-ac31df2cb1b6-config-out\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.347422 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.347415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d773150f-ac5f-4997-8240-ac31df2cb1b6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.347646 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.347623    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-web-config\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.348023 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.347977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.348120 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.348032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.348120 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.348042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d773150f-ac5f-4997-8240-ac31df2cb1b6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.348223 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.348132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.348223 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.348160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-config-volume\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.348223 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.348194    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.348989 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.348972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d773150f-ac5f-4997-8240-ac31df2cb1b6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.353644 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.353628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6zxj\" (UniqueName: \"kubernetes.io/projected/d773150f-ac5f-4997-8240-ac31df2cb1b6-kube-api-access-h6zxj\") pod \"alertmanager-main-0\" (UID: \"d773150f-ac5f-4997-8240-ac31df2cb1b6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.457858 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.457822    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:33:49.745620 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:49.745590    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:33:49.746774 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:33:49.746750    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd773150f_ac5f_4997_8240_ac31df2cb1b6.slice/crio-3f6fab558fe4fce46de815128c6713877ec455ddc43abf78f39370642dd9f1a7 WatchSource:0}: Error finding container 3f6fab558fe4fce46de815128c6713877ec455ddc43abf78f39370642dd9f1a7: Status 404 returned error can't find the container with id 3f6fab558fe4fce46de815128c6713877ec455ddc43abf78f39370642dd9f1a7
Sep 10 12:33:50.535127 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:50.535090    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d773150f-ac5f-4997-8240-ac31df2cb1b6","Type":"ContainerStarted","Data":"3f6fab558fe4fce46de815128c6713877ec455ddc43abf78f39370642dd9f1a7"}
Sep 10 12:33:51.541052 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:51.541019    2563 generic.go:356] "Generic (PLEG): container finished" podID="d773150f-ac5f-4997-8240-ac31df2cb1b6" containerID="f6603c59103f9969d6642380d410c952ebf7e2dbdc9ebf7bfd7bab8417f9b7c1" exitCode=0
Sep 10 12:33:51.541444 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:51.541103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d773150f-ac5f-4997-8240-ac31df2cb1b6","Type":"ContainerDied","Data":"f6603c59103f9969d6642380d410c952ebf7e2dbdc9ebf7bfd7bab8417f9b7c1"}
Sep 10 12:33:53.552711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:53.552620    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d773150f-ac5f-4997-8240-ac31df2cb1b6","Type":"ContainerStarted","Data":"d365c0091947ed24aa02093b4c537a4431373708180405beedc3df432cafc969"}
Sep 10 12:33:53.552711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:53.552691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d773150f-ac5f-4997-8240-ac31df2cb1b6","Type":"ContainerStarted","Data":"0646a81ebfbaf0c21fff9b43ff8ef4e9708158db6d98b93ab55395c8ef53e871"}
Sep 10 12:33:53.553101 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:53.552718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d773150f-ac5f-4997-8240-ac31df2cb1b6","Type":"ContainerStarted","Data":"8a3afc37221b855d93e0e77953798fd24a2d9cf793dd7011b0a14962cc984ce4"}
Sep 10 12:33:53.553101 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:53.552756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d773150f-ac5f-4997-8240-ac31df2cb1b6","Type":"ContainerStarted","Data":"0e7aa26e5c68274351ad487882fde8f66f8ca34b3cf34258a210d68e1e02bf9a"}
Sep 10 12:33:53.553101 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:53.552782    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d773150f-ac5f-4997-8240-ac31df2cb1b6","Type":"ContainerStarted","Data":"112cd98b8f67074fec457834de3fe39c08ddbbdb729aa867e7a06dcdb67d4ac3"}
Sep 10 12:33:53.553101 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:53.552806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d773150f-ac5f-4997-8240-ac31df2cb1b6","Type":"ContainerStarted","Data":"63eb91a8358f64f743a0b716cf993b2c6ef0b148cb63468c4b60e7d15de63e07"}
Sep 10 12:33:53.584425 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:33:53.584364    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.883221418 podStartE2EDuration="4.584341805s" totalImagesPullingTime="2.682421398s" totalInitContainerRuntime="18.698989ms" isStatefulPod=false podCreationTimestamp="2026-09-10 12:33:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:33:53.581893407 +0000 UTC m=+207.506804974" watchObservedRunningTime="2026-09-10 12:33:53.584341805 +0000 UTC m=+207.509253293"
Sep 10 12:35:26.474059 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:26.474033    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 12:35:31.963410 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:31.963367    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9fzsx"]
Sep 10 12:35:31.966585 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:31.966544    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9fzsx"
Sep 10 12:35:31.971990 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:31.971964    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 12:35:31.972717 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:31.972696    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6qh4r\""
Sep 10 12:35:31.976048 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:31.976024    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 12:35:31.997194 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:31.997156    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9fzsx"]
Sep 10 12:35:32.071581 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:32.071510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b340f02c-d532-463d-9351-33a76e39f6cb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9fzsx\" (UID: \"b340f02c-d532-463d-9351-33a76e39f6cb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fzsx"
Sep 10 12:35:32.071581 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:32.071583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvrc5\" (UniqueName: \"kubernetes.io/projected/b340f02c-d532-463d-9351-33a76e39f6cb-kube-api-access-vvrc5\") pod \"cert-manager-webhook-6486ffd997-9fzsx\" (UID: \"b340f02c-d532-463d-9351-33a76e39f6cb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fzsx"
Sep 10 12:35:32.172804 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:32.172759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b340f02c-d532-463d-9351-33a76e39f6cb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9fzsx\" (UID: \"b340f02c-d532-463d-9351-33a76e39f6cb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fzsx"
Sep 10 12:35:32.172984 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:32.172812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvrc5\" (UniqueName: \"kubernetes.io/projected/b340f02c-d532-463d-9351-33a76e39f6cb-kube-api-access-vvrc5\") pod \"cert-manager-webhook-6486ffd997-9fzsx\" (UID: \"b340f02c-d532-463d-9351-33a76e39f6cb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fzsx"
Sep 10 12:35:32.188086 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:32.188054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b340f02c-d532-463d-9351-33a76e39f6cb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9fzsx\" (UID: \"b340f02c-d532-463d-9351-33a76e39f6cb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fzsx"
Sep 10 12:35:32.189559 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:32.189530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvrc5\" (UniqueName: \"kubernetes.io/projected/b340f02c-d532-463d-9351-33a76e39f6cb-kube-api-access-vvrc5\") pod \"cert-manager-webhook-6486ffd997-9fzsx\" (UID: \"b340f02c-d532-463d-9351-33a76e39f6cb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fzsx"
Sep 10 12:35:32.286425 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:32.286333    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9fzsx"
Sep 10 12:35:32.570444 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:32.570414    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9fzsx"]
Sep 10 12:35:32.572917 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:35:32.572887    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb340f02c_d532_463d_9351_33a76e39f6cb.slice/crio-eb2a21d5164c3d8efc98844aa6920adf7e8c3384cf3dbfcaaa40fab6aa283926 WatchSource:0}: Error finding container eb2a21d5164c3d8efc98844aa6920adf7e8c3384cf3dbfcaaa40fab6aa283926: Status 404 returned error can't find the container with id eb2a21d5164c3d8efc98844aa6920adf7e8c3384cf3dbfcaaa40fab6aa283926
Sep 10 12:35:32.574976 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:32.574956    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:35:32.929799 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:32.929768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9fzsx" event={"ID":"b340f02c-d532-463d-9351-33a76e39f6cb","Type":"ContainerStarted","Data":"eb2a21d5164c3d8efc98844aa6920adf7e8c3384cf3dbfcaaa40fab6aa283926"}
Sep 10 12:35:35.945241 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:35.945206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9fzsx" event={"ID":"b340f02c-d532-463d-9351-33a76e39f6cb","Type":"ContainerStarted","Data":"eeb25575152d750810441408bb362c84bbaa68b32876959f47f1d1c2f6211b70"}
Sep 10 12:35:35.945700 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:35.945374    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9fzsx"
Sep 10 12:35:35.968499 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:35.968449    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-9fzsx" podStartSLOduration=1.704138608 podStartE2EDuration="4.968433192s" totalImagesPullingTime="3.264294584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:35:35.968098842 +0000 UTC m=+309.893010332" watchObservedRunningTime="2026-09-10 12:35:35.968433192 +0000 UTC m=+309.893344679"
Sep 10 12:35:41.952637 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:41.952599    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9fzsx"
Sep 10 12:35:50.873431 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:50.873397    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv"]
Sep 10 12:35:50.876716 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:50.876692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv"
Sep 10 12:35:50.879144 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:50.879124    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:35:50.879986 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:50.879964    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 12:35:50.880102 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:50.879987    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-8sm6f\""
Sep 10 12:35:50.886007 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:50.885980    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv"]
Sep 10 12:35:51.031183 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:51.031132    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44nb\" (UniqueName: \"kubernetes.io/projected/1de3f6dc-4b41-43a5-b0fa-127f68713c57-kube-api-access-g44nb\") pod \"openshift-lws-operator-fd8ccff4c-nhzlv\" (UID: \"1de3f6dc-4b41-43a5-b0fa-127f68713c57\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv"
Sep 10 12:35:51.031183 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:51.031183    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1de3f6dc-4b41-43a5-b0fa-127f68713c57-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nhzlv\" (UID: \"1de3f6dc-4b41-43a5-b0fa-127f68713c57\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv"
Sep 10 12:35:51.132685 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:51.132590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44nb\" (UniqueName: \"kubernetes.io/projected/1de3f6dc-4b41-43a5-b0fa-127f68713c57-kube-api-access-g44nb\") pod \"openshift-lws-operator-fd8ccff4c-nhzlv\" (UID: \"1de3f6dc-4b41-43a5-b0fa-127f68713c57\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv"
Sep 10 12:35:51.132685 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:51.132640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1de3f6dc-4b41-43a5-b0fa-127f68713c57-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nhzlv\" (UID: \"1de3f6dc-4b41-43a5-b0fa-127f68713c57\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv"
Sep 10 12:35:51.133065 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:51.133042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1de3f6dc-4b41-43a5-b0fa-127f68713c57-tmp\") pod \"openshift-lws-operator-fd8ccff4c-nhzlv\" (UID: \"1de3f6dc-4b41-43a5-b0fa-127f68713c57\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv"
Sep 10 12:35:51.141601 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:51.141542    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44nb\" (UniqueName: \"kubernetes.io/projected/1de3f6dc-4b41-43a5-b0fa-127f68713c57-kube-api-access-g44nb\") pod \"openshift-lws-operator-fd8ccff4c-nhzlv\" (UID: \"1de3f6dc-4b41-43a5-b0fa-127f68713c57\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv"
Sep 10 12:35:51.186292 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:51.186252    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv"
Sep 10 12:35:51.457954 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:51.457914    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv"]
Sep 10 12:35:51.461046 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:35:51.461021    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1de3f6dc_4b41_43a5_b0fa_127f68713c57.slice/crio-863cc258050a59702122052f60902a339d1472658e2b5be7ecb85b1e3fd67715 WatchSource:0}: Error finding container 863cc258050a59702122052f60902a339d1472658e2b5be7ecb85b1e3fd67715: Status 404 returned error can't find the container with id 863cc258050a59702122052f60902a339d1472658e2b5be7ecb85b1e3fd67715
Sep 10 12:35:52.010364 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:35:52.010332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv" event={"ID":"1de3f6dc-4b41-43a5-b0fa-127f68713c57","Type":"ContainerStarted","Data":"863cc258050a59702122052f60902a339d1472658e2b5be7ecb85b1e3fd67715"}
Sep 10 12:36:04.062623 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:04.062580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv" event={"ID":"1de3f6dc-4b41-43a5-b0fa-127f68713c57","Type":"ContainerStarted","Data":"d161d059a0d0842a88cd1d13598451a8a3c23da5f74cd930ffb3fb6614fb6ce8"}
Sep 10 12:36:04.086110 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:04.086055    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-nhzlv" podStartSLOduration=1.897101437 podStartE2EDuration="14.086039726s" totalImagesPullingTime="12.188938289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:36:04.083838292 +0000 UTC m=+338.008749779" watchObservedRunningTime="2026-09-10 12:36:04.086039726 +0000 UTC m=+338.010951214"
Sep 10 12:36:22.867432 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.867356    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"]
Sep 10 12:36:22.869844 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.869827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.873648 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.873620    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 12:36:22.873815 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.873692    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 12:36:22.873944 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.873873    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 12:36:22.873944 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.873909    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-j45z8\""
Sep 10 12:36:22.874098 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.874083    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 12:36:22.889740 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.889710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk9sj\" (UniqueName: \"kubernetes.io/projected/48f3d0a0-8c15-4ffe-a104-34fa94267f47-kube-api-access-nk9sj\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.889896 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.889758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48f3d0a0-8c15-4ffe-a104-34fa94267f47-tmp\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.889896 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.889782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48f3d0a0-8c15-4ffe-a104-34fa94267f47-manifests\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.889896 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.889841    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48f3d0a0-8c15-4ffe-a104-34fa94267f47-apiservice-cert\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.889896 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.889883    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48f3d0a0-8c15-4ffe-a104-34fa94267f47-webhook-cert\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.897662 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.897638    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"]
Sep 10 12:36:22.991317 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.991280    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48f3d0a0-8c15-4ffe-a104-34fa94267f47-apiservice-cert\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.991521 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.991324    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48f3d0a0-8c15-4ffe-a104-34fa94267f47-webhook-cert\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.991521 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.991363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk9sj\" (UniqueName: \"kubernetes.io/projected/48f3d0a0-8c15-4ffe-a104-34fa94267f47-kube-api-access-nk9sj\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.991521 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.991414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48f3d0a0-8c15-4ffe-a104-34fa94267f47-tmp\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.991521 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.991446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48f3d0a0-8c15-4ffe-a104-34fa94267f47-manifests\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.991903 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.991880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48f3d0a0-8c15-4ffe-a104-34fa94267f47-manifests\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.993741 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.993710    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48f3d0a0-8c15-4ffe-a104-34fa94267f47-tmp\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.993983 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.993962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48f3d0a0-8c15-4ffe-a104-34fa94267f47-webhook-cert\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:22.994053 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:22.993962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48f3d0a0-8c15-4ffe-a104-34fa94267f47-apiservice-cert\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:23.009160 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:23.009135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk9sj\" (UniqueName: \"kubernetes.io/projected/48f3d0a0-8c15-4ffe-a104-34fa94267f47-kube-api-access-nk9sj\") pod \"opendatahub-operator-controller-manager-669b9dff44-5cfwp\" (UID: \"48f3d0a0-8c15-4ffe-a104-34fa94267f47\") " pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:23.180387 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:23.180356    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:23.466820 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:23.466785    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"]
Sep 10 12:36:23.468641 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:36:23.468616    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f3d0a0_8c15_4ffe_a104_34fa94267f47.slice/crio-e69a2fc85dff08fc79d55a007fe3d9bc3e1ce6cb92caaaf6c35036d59c00b2b1 WatchSource:0}: Error finding container e69a2fc85dff08fc79d55a007fe3d9bc3e1ce6cb92caaaf6c35036d59c00b2b1: Status 404 returned error can't find the container with id e69a2fc85dff08fc79d55a007fe3d9bc3e1ce6cb92caaaf6c35036d59c00b2b1
Sep 10 12:36:24.142455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:24.142417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp" event={"ID":"48f3d0a0-8c15-4ffe-a104-34fa94267f47","Type":"ContainerStarted","Data":"e69a2fc85dff08fc79d55a007fe3d9bc3e1ce6cb92caaaf6c35036d59c00b2b1"}
Sep 10 12:36:27.158501 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:27.158461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp" event={"ID":"48f3d0a0-8c15-4ffe-a104-34fa94267f47","Type":"ContainerStarted","Data":"2d36255139e28e226ae39904af672a24fb8a56b3659517c815df3d4ddedbf226"}
Sep 10 12:36:28.164842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:28.164795    2563 generic.go:356] "Generic (PLEG): container finished" podID="48f3d0a0-8c15-4ffe-a104-34fa94267f47" containerID="2d36255139e28e226ae39904af672a24fb8a56b3659517c815df3d4ddedbf226" exitCode=0
Sep 10 12:36:28.165263 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:28.164878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp" event={"ID":"48f3d0a0-8c15-4ffe-a104-34fa94267f47","Type":"ContainerDied","Data":"2d36255139e28e226ae39904af672a24fb8a56b3659517c815df3d4ddedbf226"}
Sep 10 12:36:29.170916 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:29.170881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp" event={"ID":"48f3d0a0-8c15-4ffe-a104-34fa94267f47","Type":"ContainerStarted","Data":"f145fa36d3e01109af7cea8d31325185babd25e69d7168c8ed1cf362542e1b1e"}
Sep 10 12:36:29.171317 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:29.171059    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:29.196130 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:29.196080    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp" podStartSLOduration=2.744169667 podStartE2EDuration="7.196064757s" totalImagesPullingTime="3.372228993s" totalInitContainerRuntime="1.079666097s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:36:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:36:29.19413426 +0000 UTC m=+363.119045750" watchObservedRunningTime="2026-09-10 12:36:29.196064757 +0000 UTC m=+363.120976245"
Sep 10 12:36:40.178925 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:40.178886    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-669b9dff44-5cfwp"
Sep 10 12:36:42.457455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.457416    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"]
Sep 10 12:36:42.465960 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.465936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"
Sep 10 12:36:42.471472 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.471447    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c2qxn\""
Sep 10 12:36:42.472263 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.472236    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 12:36:42.472399 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.472297    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 12:36:42.474692 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.474673    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 12:36:42.477828 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.477808    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 12:36:42.488281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.488252    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"]
Sep 10 12:36:42.567428 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.567388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmcm4\" (UniqueName: \"kubernetes.io/projected/6fd4b376-afab-4151-a2a8-1dd23c901612-kube-api-access-bmcm4\") pod \"kube-auth-proxy-669df89dc4-gzv5p\" (UID: \"6fd4b376-afab-4151-a2a8-1dd23c901612\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"
Sep 10 12:36:42.567428 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.567426    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fd4b376-afab-4151-a2a8-1dd23c901612-tmp\") pod \"kube-auth-proxy-669df89dc4-gzv5p\" (UID: \"6fd4b376-afab-4151-a2a8-1dd23c901612\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"
Sep 10 12:36:42.567684 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.567465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd4b376-afab-4151-a2a8-1dd23c901612-tls-certs\") pod \"kube-auth-proxy-669df89dc4-gzv5p\" (UID: \"6fd4b376-afab-4151-a2a8-1dd23c901612\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"
Sep 10 12:36:42.668702 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.668662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmcm4\" (UniqueName: \"kubernetes.io/projected/6fd4b376-afab-4151-a2a8-1dd23c901612-kube-api-access-bmcm4\") pod \"kube-auth-proxy-669df89dc4-gzv5p\" (UID: \"6fd4b376-afab-4151-a2a8-1dd23c901612\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"
Sep 10 12:36:42.668702 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.668702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fd4b376-afab-4151-a2a8-1dd23c901612-tmp\") pod \"kube-auth-proxy-669df89dc4-gzv5p\" (UID: \"6fd4b376-afab-4151-a2a8-1dd23c901612\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"
Sep 10 12:36:42.668907 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.668729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd4b376-afab-4151-a2a8-1dd23c901612-tls-certs\") pod \"kube-auth-proxy-669df89dc4-gzv5p\" (UID: \"6fd4b376-afab-4151-a2a8-1dd23c901612\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"
Sep 10 12:36:42.671022 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.670992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6fd4b376-afab-4151-a2a8-1dd23c901612-tmp\") pod \"kube-auth-proxy-669df89dc4-gzv5p\" (UID: \"6fd4b376-afab-4151-a2a8-1dd23c901612\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"
Sep 10 12:36:42.671129 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.671045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd4b376-afab-4151-a2a8-1dd23c901612-tls-certs\") pod \"kube-auth-proxy-669df89dc4-gzv5p\" (UID: \"6fd4b376-afab-4151-a2a8-1dd23c901612\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"
Sep 10 12:36:42.678663 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.678640    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmcm4\" (UniqueName: \"kubernetes.io/projected/6fd4b376-afab-4151-a2a8-1dd23c901612-kube-api-access-bmcm4\") pod \"kube-auth-proxy-669df89dc4-gzv5p\" (UID: \"6fd4b376-afab-4151-a2a8-1dd23c901612\") " pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"
Sep 10 12:36:42.775276 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:42.775195    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"
Sep 10 12:36:43.047109 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:43.047083    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p"]
Sep 10 12:36:43.049160 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:36:43.049134    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd4b376_afab_4151_a2a8_1dd23c901612.slice/crio-94ce64663524a8cb2f821388ae3ab6cc7f71a93f1e3788724bfc5239e38b4556 WatchSource:0}: Error finding container 94ce64663524a8cb2f821388ae3ab6cc7f71a93f1e3788724bfc5239e38b4556: Status 404 returned error can't find the container with id 94ce64663524a8cb2f821388ae3ab6cc7f71a93f1e3788724bfc5239e38b4556
Sep 10 12:36:43.233925 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:43.233890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p" event={"ID":"6fd4b376-afab-4151-a2a8-1dd23c901612","Type":"ContainerStarted","Data":"94ce64663524a8cb2f821388ae3ab6cc7f71a93f1e3788724bfc5239e38b4556"}
Sep 10 12:36:44.175382 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.175343    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"]
Sep 10 12:36:44.178745 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.178697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.181221 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.181191    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 12:36:44.181221 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.181215    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 12:36:44.181440 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.181301    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mgxvn\""
Sep 10 12:36:44.181440 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.181196    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 12:36:44.181526 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.181197    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 12:36:44.193230 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.193182    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"]
Sep 10 12:36:44.282341 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.282300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87dd9487-fb1c-429e-9f0e-7004990b078b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.282539 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.282360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.282539 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.282386    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.282539 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.282497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.282703 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.282544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.282703 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.282621    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.282703 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.282659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsv4m\" (UniqueName: \"kubernetes.io/projected/87dd9487-fb1c-429e-9f0e-7004990b078b-kube-api-access-bsv4m\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.383529 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.383487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsv4m\" (UniqueName: \"kubernetes.io/projected/87dd9487-fb1c-429e-9f0e-7004990b078b-kube-api-access-bsv4m\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.383731 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.383688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87dd9487-fb1c-429e-9f0e-7004990b078b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.383731 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.383725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.383856 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.383755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.383856 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.383808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.383856 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.383845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.384000 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.383886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.384530 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.384481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.386706 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.386675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.386830 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.386788    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.386830 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.386810    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87dd9487-fb1c-429e-9f0e-7004990b078b-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.386914 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.386893    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.394359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.394309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.395030 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.395006    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsv4m\" (UniqueName: \"kubernetes.io/projected/87dd9487-fb1c-429e-9f0e-7004990b078b-kube-api-access-bsv4m\") pod \"istiod-openshift-gateway-7b7995d5cd-mzlqs\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.493173 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.493075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:44.784679 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:44.784648    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"]
Sep 10 12:36:45.290336 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:36:45.290298    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87dd9487_fb1c_429e_9f0e_7004990b078b.slice/crio-c34787481bd52e2dba602daaea286365465307ebd51e4a2ac55a220a1d18f30b WatchSource:0}: Error finding container c34787481bd52e2dba602daaea286365465307ebd51e4a2ac55a220a1d18f30b: Status 404 returned error can't find the container with id c34787481bd52e2dba602daaea286365465307ebd51e4a2ac55a220a1d18f30b
Sep 10 12:36:46.257226 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:46.257126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p" event={"ID":"6fd4b376-afab-4151-a2a8-1dd23c901612","Type":"ContainerStarted","Data":"d157eaaf7259ff9af978d1382adb4d333b50589e627a42e892f484abf433d672"}
Sep 10 12:36:46.258389 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:46.258362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs" event={"ID":"87dd9487-fb1c-429e-9f0e-7004990b078b","Type":"ContainerStarted","Data":"c34787481bd52e2dba602daaea286365465307ebd51e4a2ac55a220a1d18f30b"}
Sep 10 12:36:46.277648 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:46.277593    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-669df89dc4-gzv5p" podStartSLOduration=1.35964763 podStartE2EDuration="4.277577015s" totalImagesPullingTime="2.917929385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:36:46.276010308 +0000 UTC m=+380.200921788" watchObservedRunningTime="2026-09-10 12:36:46.277577015 +0000 UTC m=+380.202488496"
Sep 10 12:36:49.242331 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:49.242285    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:36:49.242635 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:49.242365    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:36:50.278118 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:50.278085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs" event={"ID":"87dd9487-fb1c-429e-9f0e-7004990b078b","Type":"ContainerStarted","Data":"445fa586bbc8e60998561d610ce82947ac0e47d2ef7a9ff5cc700004fba215b5"}
Sep 10 12:36:50.278537 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:50.278234    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:50.279816 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:50.279787    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-mzlqs container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 12:36:50.279941 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:50.279838    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs" podUID="87dd9487-fb1c-429e-9f0e-7004990b078b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 12:36:50.310223 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:50.310171    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs" podStartSLOduration=2.360580375 podStartE2EDuration="6.31015252s" totalImagesPullingTime="3.949572145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:36:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:36:50.309736731 +0000 UTC m=+384.234648220" watchObservedRunningTime="2026-09-10 12:36:50.31015252 +0000 UTC m=+384.235064008"
Sep 10 12:36:51.283567 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:51.283521    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:36:53.301882 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:53.301847    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jxq4n"]
Sep 10 12:36:53.304575 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:53.304540    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jxq4n"
Sep 10 12:36:53.307190 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:53.307164    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 12:36:53.307512 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:53.307493    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-6hk5j\""
Sep 10 12:36:53.315692 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:53.315668    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jxq4n"]
Sep 10 12:36:53.473039 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:53.473000    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f31c4d-8948-4332-bc14-40e5a0968652-tls-certs\") pod \"model-serving-api-54d89bd5fc-jxq4n\" (UID: \"05f31c4d-8948-4332-bc14-40e5a0968652\") " pod="opendatahub/model-serving-api-54d89bd5fc-jxq4n"
Sep 10 12:36:53.473039 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:53.473051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4bj7\" (UniqueName: \"kubernetes.io/projected/05f31c4d-8948-4332-bc14-40e5a0968652-kube-api-access-r4bj7\") pod \"model-serving-api-54d89bd5fc-jxq4n\" (UID: \"05f31c4d-8948-4332-bc14-40e5a0968652\") " pod="opendatahub/model-serving-api-54d89bd5fc-jxq4n"
Sep 10 12:36:53.574396 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:53.574305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f31c4d-8948-4332-bc14-40e5a0968652-tls-certs\") pod \"model-serving-api-54d89bd5fc-jxq4n\" (UID: \"05f31c4d-8948-4332-bc14-40e5a0968652\") " pod="opendatahub/model-serving-api-54d89bd5fc-jxq4n"
Sep 10 12:36:53.574396 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:53.574355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4bj7\" (UniqueName: \"kubernetes.io/projected/05f31c4d-8948-4332-bc14-40e5a0968652-kube-api-access-r4bj7\") pod \"model-serving-api-54d89bd5fc-jxq4n\" (UID: \"05f31c4d-8948-4332-bc14-40e5a0968652\") " pod="opendatahub/model-serving-api-54d89bd5fc-jxq4n"
Sep 10 12:36:53.576747 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:53.576728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05f31c4d-8948-4332-bc14-40e5a0968652-tls-certs\") pod \"model-serving-api-54d89bd5fc-jxq4n\" (UID: \"05f31c4d-8948-4332-bc14-40e5a0968652\") " pod="opendatahub/model-serving-api-54d89bd5fc-jxq4n"
Sep 10 12:36:53.582334 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:53.582313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4bj7\" (UniqueName: \"kubernetes.io/projected/05f31c4d-8948-4332-bc14-40e5a0968652-kube-api-access-r4bj7\") pod \"model-serving-api-54d89bd5fc-jxq4n\" (UID: \"05f31c4d-8948-4332-bc14-40e5a0968652\") " pod="opendatahub/model-serving-api-54d89bd5fc-jxq4n"
Sep 10 12:36:53.615674 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:53.615637    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-jxq4n"
Sep 10 12:36:53.886578 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:53.886537    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-jxq4n"]
Sep 10 12:36:53.888981 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:36:53.888957    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f31c4d_8948_4332_bc14_40e5a0968652.slice/crio-e44526ec39d2dc8bf751b28dea2af2e27c546ba942b1ec05ae7026f1fb5f47f7 WatchSource:0}: Error finding container e44526ec39d2dc8bf751b28dea2af2e27c546ba942b1ec05ae7026f1fb5f47f7: Status 404 returned error can't find the container with id e44526ec39d2dc8bf751b28dea2af2e27c546ba942b1ec05ae7026f1fb5f47f7
Sep 10 12:36:54.297311 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:54.297276    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jxq4n" event={"ID":"05f31c4d-8948-4332-bc14-40e5a0968652","Type":"ContainerStarted","Data":"e44526ec39d2dc8bf751b28dea2af2e27c546ba942b1ec05ae7026f1fb5f47f7"}
Sep 10 12:36:56.308066 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:56.308029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-jxq4n" event={"ID":"05f31c4d-8948-4332-bc14-40e5a0968652","Type":"ContainerStarted","Data":"79b603461399b25c8460bd4fd81c8cee4c6cfb60d9c67b945f4436ceab9abb00"}
Sep 10 12:36:56.308439 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:56.308106    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-jxq4n"
Sep 10 12:36:56.330743 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:36:56.330693    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-jxq4n" podStartSLOduration=1.633927248 podStartE2EDuration="3.330677365s" totalImagesPullingTime="1.696750117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:36:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:36:56.330289493 +0000 UTC m=+390.255200983" watchObservedRunningTime="2026-09-10 12:36:56.330677365 +0000 UTC m=+390.255588854"
Sep 10 12:37:03.155078 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.155041    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw"]
Sep 10 12:37:03.163372 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.163351    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw"
Sep 10 12:37:03.165900 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.165877    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 12:37:03.167021 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.167003    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9zrps\""
Sep 10 12:37:03.177057 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.177034    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj"]
Sep 10 12:37:03.179802 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.179782    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw"]
Sep 10 12:37:03.179898 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.179885    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj"
Sep 10 12:37:03.182198 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.182175    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hptzn\""
Sep 10 12:37:03.182423 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.182398    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 12:37:03.192964 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.192941    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj"]
Sep 10 12:37:03.258079 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.258046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87452133-eecf-4359-b5bd-a7b37c5750c8-cert\") pod \"kserve-controller-manager-d6f4b65b8-jx7nw\" (UID: \"87452133-eecf-4359-b5bd-a7b37c5750c8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw"
Sep 10 12:37:03.258270 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.258107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c7804bb-3e5c-4105-9ab1-262b3a349561-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rpnjj\" (UID: \"2c7804bb-3e5c-4105-9ab1-262b3a349561\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj"
Sep 10 12:37:03.258270 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.258128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5st7d\" (UniqueName: \"kubernetes.io/projected/87452133-eecf-4359-b5bd-a7b37c5750c8-kube-api-access-5st7d\") pod \"kserve-controller-manager-d6f4b65b8-jx7nw\" (UID: \"87452133-eecf-4359-b5bd-a7b37c5750c8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw"
Sep 10 12:37:03.258270 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.258165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x4dv\" (UniqueName: \"kubernetes.io/projected/2c7804bb-3e5c-4105-9ab1-262b3a349561-kube-api-access-7x4dv\") pod \"llmisvc-controller-manager-777c6bd7d5-rpnjj\" (UID: \"2c7804bb-3e5c-4105-9ab1-262b3a349561\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj"
Sep 10 12:37:03.358620 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.358590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87452133-eecf-4359-b5bd-a7b37c5750c8-cert\") pod \"kserve-controller-manager-d6f4b65b8-jx7nw\" (UID: \"87452133-eecf-4359-b5bd-a7b37c5750c8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw"
Sep 10 12:37:03.358805 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.358687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c7804bb-3e5c-4105-9ab1-262b3a349561-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rpnjj\" (UID: \"2c7804bb-3e5c-4105-9ab1-262b3a349561\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj"
Sep 10 12:37:03.358805 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.358712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5st7d\" (UniqueName: \"kubernetes.io/projected/87452133-eecf-4359-b5bd-a7b37c5750c8-kube-api-access-5st7d\") pod \"kserve-controller-manager-d6f4b65b8-jx7nw\" (UID: \"87452133-eecf-4359-b5bd-a7b37c5750c8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw"
Sep 10 12:37:03.358805 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.358736    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x4dv\" (UniqueName: \"kubernetes.io/projected/2c7804bb-3e5c-4105-9ab1-262b3a349561-kube-api-access-7x4dv\") pod \"llmisvc-controller-manager-777c6bd7d5-rpnjj\" (UID: \"2c7804bb-3e5c-4105-9ab1-262b3a349561\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj"
Sep 10 12:37:03.361229 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.361204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c7804bb-3e5c-4105-9ab1-262b3a349561-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-rpnjj\" (UID: \"2c7804bb-3e5c-4105-9ab1-262b3a349561\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj"
Sep 10 12:37:03.361330 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.361303    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87452133-eecf-4359-b5bd-a7b37c5750c8-cert\") pod \"kserve-controller-manager-d6f4b65b8-jx7nw\" (UID: \"87452133-eecf-4359-b5bd-a7b37c5750c8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw"
Sep 10 12:37:03.367074 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.367051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x4dv\" (UniqueName: \"kubernetes.io/projected/2c7804bb-3e5c-4105-9ab1-262b3a349561-kube-api-access-7x4dv\") pod \"llmisvc-controller-manager-777c6bd7d5-rpnjj\" (UID: \"2c7804bb-3e5c-4105-9ab1-262b3a349561\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj"
Sep 10 12:37:03.367492 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.367475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5st7d\" (UniqueName: \"kubernetes.io/projected/87452133-eecf-4359-b5bd-a7b37c5750c8-kube-api-access-5st7d\") pod \"kserve-controller-manager-d6f4b65b8-jx7nw\" (UID: \"87452133-eecf-4359-b5bd-a7b37c5750c8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw"
Sep 10 12:37:03.475032 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.475001    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw"
Sep 10 12:37:03.492375 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.492352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj"
Sep 10 12:37:03.773666 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.773594    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw"]
Sep 10 12:37:03.777049 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:03.777024    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj"]
Sep 10 12:37:03.777896 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:37:03.777874    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87452133_eecf_4359_b5bd_a7b37c5750c8.slice/crio-5705a2344ccc1926e5d0eddbf908698cb50b487373a944329a865d6dc2021110 WatchSource:0}: Error finding container 5705a2344ccc1926e5d0eddbf908698cb50b487373a944329a865d6dc2021110: Status 404 returned error can't find the container with id 5705a2344ccc1926e5d0eddbf908698cb50b487373a944329a865d6dc2021110
Sep 10 12:37:03.778217 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:37:03.778195    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2c7804bb_3e5c_4105_9ab1_262b3a349561.slice/crio-217e3611a54553455a9687266e02c3474849511ff1b8aa4b741104b3cc7c6da3 WatchSource:0}: Error finding container 217e3611a54553455a9687266e02c3474849511ff1b8aa4b741104b3cc7c6da3: Status 404 returned error can't find the container with id 217e3611a54553455a9687266e02c3474849511ff1b8aa4b741104b3cc7c6da3
Sep 10 12:37:04.127432 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.127349    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"]
Sep 10 12:37:04.127613 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.127534    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs" podUID="87dd9487-fb1c-429e-9f0e-7004990b078b" containerName="discovery" containerID="cri-o://445fa586bbc8e60998561d610ce82947ac0e47d2ef7a9ff5cc700004fba215b5" gracePeriod=30
Sep 10 12:37:04.344985 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.344945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw" event={"ID":"87452133-eecf-4359-b5bd-a7b37c5750c8","Type":"ContainerStarted","Data":"5705a2344ccc1926e5d0eddbf908698cb50b487373a944329a865d6dc2021110"}
Sep 10 12:37:04.349394 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.349348    2563 generic.go:356] "Generic (PLEG): container finished" podID="87dd9487-fb1c-429e-9f0e-7004990b078b" containerID="445fa586bbc8e60998561d610ce82947ac0e47d2ef7a9ff5cc700004fba215b5" exitCode=0
Sep 10 12:37:04.349518 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.349465    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs" event={"ID":"87dd9487-fb1c-429e-9f0e-7004990b078b","Type":"ContainerDied","Data":"445fa586bbc8e60998561d610ce82947ac0e47d2ef7a9ff5cc700004fba215b5"}
Sep 10 12:37:04.355173 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.355128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj" event={"ID":"2c7804bb-3e5c-4105-9ab1-262b3a349561","Type":"ContainerStarted","Data":"217e3611a54553455a9687266e02c3474849511ff1b8aa4b741104b3cc7c6da3"}
Sep 10 12:37:04.363709 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.363687    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:37:04.365623 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.365597    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-csr-ca-configmap\") pod \"87dd9487-fb1c-429e-9f0e-7004990b078b\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") "
Sep 10 12:37:04.365728 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.365643    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87dd9487-fb1c-429e-9f0e-7004990b078b-local-certs\") pod \"87dd9487-fb1c-429e-9f0e-7004990b078b\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") "
Sep 10 12:37:04.365728 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.365671    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-kubeconfig\") pod \"87dd9487-fb1c-429e-9f0e-7004990b078b\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") "
Sep 10 12:37:04.365837 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.365727    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsv4m\" (UniqueName: \"kubernetes.io/projected/87dd9487-fb1c-429e-9f0e-7004990b078b-kube-api-access-bsv4m\") pod \"87dd9487-fb1c-429e-9f0e-7004990b078b\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") "
Sep 10 12:37:04.365837 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.365756    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-cacerts\") pod \"87dd9487-fb1c-429e-9f0e-7004990b078b\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") "
Sep 10 12:37:04.365837 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.365788    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-csr-dns-cert\") pod \"87dd9487-fb1c-429e-9f0e-7004990b078b\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") "
Sep 10 12:37:04.365990 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.365842    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-token\") pod \"87dd9487-fb1c-429e-9f0e-7004990b078b\" (UID: \"87dd9487-fb1c-429e-9f0e-7004990b078b\") "
Sep 10 12:37:04.365990 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.365975    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "87dd9487-fb1c-429e-9f0e-7004990b078b" (UID: "87dd9487-fb1c-429e-9f0e-7004990b078b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:37:04.366166 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.366087    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-csr-ca-configmap\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:37:04.369895 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.369867    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87dd9487-fb1c-429e-9f0e-7004990b078b-kube-api-access-bsv4m" (OuterVolumeSpecName: "kube-api-access-bsv4m") pod "87dd9487-fb1c-429e-9f0e-7004990b078b" (UID: "87dd9487-fb1c-429e-9f0e-7004990b078b"). InnerVolumeSpecName "kube-api-access-bsv4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:37:04.369995 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.369952    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-cacerts" (OuterVolumeSpecName: "cacerts") pod "87dd9487-fb1c-429e-9f0e-7004990b078b" (UID: "87dd9487-fb1c-429e-9f0e-7004990b078b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:37:04.370107 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.370083    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87dd9487-fb1c-429e-9f0e-7004990b078b-local-certs" (OuterVolumeSpecName: "local-certs") pod "87dd9487-fb1c-429e-9f0e-7004990b078b" (UID: "87dd9487-fb1c-429e-9f0e-7004990b078b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:37:04.370480 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.370454    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-token" (OuterVolumeSpecName: "istio-token") pod "87dd9487-fb1c-429e-9f0e-7004990b078b" (UID: "87dd9487-fb1c-429e-9f0e-7004990b078b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:37:04.370913 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.370881    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "87dd9487-fb1c-429e-9f0e-7004990b078b" (UID: "87dd9487-fb1c-429e-9f0e-7004990b078b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:37:04.372340 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.372315    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "87dd9487-fb1c-429e-9f0e-7004990b078b" (UID: "87dd9487-fb1c-429e-9f0e-7004990b078b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:37:04.466481 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.466451    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-token\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:37:04.466481 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.466479    2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/87dd9487-fb1c-429e-9f0e-7004990b078b-local-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:37:04.466724 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.466490    2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-kubeconfig\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:37:04.466724 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.466503    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsv4m\" (UniqueName: \"kubernetes.io/projected/87dd9487-fb1c-429e-9f0e-7004990b078b-kube-api-access-bsv4m\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:37:04.466724 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.466516    2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-cacerts\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:37:04.466724 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:04.466527    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/87dd9487-fb1c-429e-9f0e-7004990b078b-istio-csr-dns-cert\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:37:05.361978 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:05.361940    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"
Sep 10 12:37:05.361978 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:05.361957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs" event={"ID":"87dd9487-fb1c-429e-9f0e-7004990b078b","Type":"ContainerDied","Data":"c34787481bd52e2dba602daaea286365465307ebd51e4a2ac55a220a1d18f30b"}
Sep 10 12:37:05.362449 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:05.362024    2563 scope.go:117] "RemoveContainer" containerID="445fa586bbc8e60998561d610ce82947ac0e47d2ef7a9ff5cc700004fba215b5"
Sep 10 12:37:05.381399 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:05.381369    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"]
Sep 10 12:37:05.384971 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:05.384947    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mzlqs"]
Sep 10 12:37:06.704833 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:06.704793    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87dd9487-fb1c-429e-9f0e-7004990b078b" path="/var/lib/kubelet/pods/87dd9487-fb1c-429e-9f0e-7004990b078b/volumes"
Sep 10 12:37:07.317069 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:07.317035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-jxq4n"
Sep 10 12:37:07.376955 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:07.376914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw" event={"ID":"87452133-eecf-4359-b5bd-a7b37c5750c8","Type":"ContainerStarted","Data":"dd08c089f7971bda63b3f9d5ceca85040b620576032ef369de0a467636ff3929"}
Sep 10 12:37:07.377153 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:07.377011    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw"
Sep 10 12:37:07.379800 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:07.379773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj" event={"ID":"2c7804bb-3e5c-4105-9ab1-262b3a349561","Type":"ContainerStarted","Data":"1db46bbe9916d9dc4a329ae20f4102011140841ed5272185c4d2068e658768e3"}
Sep 10 12:37:07.379919 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:07.379841    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj"
Sep 10 12:37:07.415711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:07.415659    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw" podStartSLOduration=1.704429036 podStartE2EDuration="4.415642072s" totalImagesPullingTime="2.711213036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:37:07.414610494 +0000 UTC m=+401.339521982" watchObservedRunningTime="2026-09-10 12:37:07.415642072 +0000 UTC m=+401.340553560"
Sep 10 12:37:07.467710 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:07.467654    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj" podStartSLOduration=1.712734771 podStartE2EDuration="4.467638362s" totalImagesPullingTime="2.754903591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:37:07.457352694 +0000 UTC m=+401.382264181" watchObservedRunningTime="2026-09-10 12:37:07.467638362 +0000 UTC m=+401.392549850"
Sep 10 12:37:38.387888 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:38.387856    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-rpnjj"
Sep 10 12:37:38.390896 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:37:38.390873    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-jx7nw"
Sep 10 12:38:14.778949 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:14.778910    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bgjwx"]
Sep 10 12:38:14.779381 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:14.779275    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87dd9487-fb1c-429e-9f0e-7004990b078b" containerName="discovery"
Sep 10 12:38:14.779381 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:14.779288    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="87dd9487-fb1c-429e-9f0e-7004990b078b" containerName="discovery"
Sep 10 12:38:14.779381 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:14.779355    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="87dd9487-fb1c-429e-9f0e-7004990b078b" containerName="discovery"
Sep 10 12:38:14.781461 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:14.781445    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-bgjwx"
Sep 10 12:38:14.784342 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:14.784322    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-q8vtf\""
Sep 10 12:38:14.784870 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:14.784851    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 12:38:14.784977 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:14.784921    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 12:38:14.792736 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:14.792716    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bgjwx"]
Sep 10 12:38:14.842312 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:14.842277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm92g\" (UniqueName: \"kubernetes.io/projected/aa9b2d36-210f-4c0f-82c4-cb5a233c29ef-kube-api-access-tm92g\") pod \"authorino-operator-76b478bd78-bgjwx\" (UID: \"aa9b2d36-210f-4c0f-82c4-cb5a233c29ef\") " pod="kuadrant-system/authorino-operator-76b478bd78-bgjwx"
Sep 10 12:38:14.942895 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:14.942856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm92g\" (UniqueName: \"kubernetes.io/projected/aa9b2d36-210f-4c0f-82c4-cb5a233c29ef-kube-api-access-tm92g\") pod \"authorino-operator-76b478bd78-bgjwx\" (UID: \"aa9b2d36-210f-4c0f-82c4-cb5a233c29ef\") " pod="kuadrant-system/authorino-operator-76b478bd78-bgjwx"
Sep 10 12:38:14.962894 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:14.962863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm92g\" (UniqueName: \"kubernetes.io/projected/aa9b2d36-210f-4c0f-82c4-cb5a233c29ef-kube-api-access-tm92g\") pod \"authorino-operator-76b478bd78-bgjwx\" (UID: \"aa9b2d36-210f-4c0f-82c4-cb5a233c29ef\") " pod="kuadrant-system/authorino-operator-76b478bd78-bgjwx"
Sep 10 12:38:15.091733 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:15.091643    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-bgjwx"
Sep 10 12:38:15.380626 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:15.380536    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bgjwx"]
Sep 10 12:38:15.383694 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:38:15.383670    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9b2d36_210f_4c0f_82c4_cb5a233c29ef.slice/crio-1b6c6f44ad46a32041ca1cda75c82f3654f8c05d73b8214af3d821965156ec3d WatchSource:0}: Error finding container 1b6c6f44ad46a32041ca1cda75c82f3654f8c05d73b8214af3d821965156ec3d: Status 404 returned error can't find the container with id 1b6c6f44ad46a32041ca1cda75c82f3654f8c05d73b8214af3d821965156ec3d
Sep 10 12:38:15.694273 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:15.694238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-bgjwx" event={"ID":"aa9b2d36-210f-4c0f-82c4-cb5a233c29ef","Type":"ContainerStarted","Data":"1b6c6f44ad46a32041ca1cda75c82f3654f8c05d73b8214af3d821965156ec3d"}
Sep 10 12:38:18.715194 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:18.715149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-bgjwx" event={"ID":"aa9b2d36-210f-4c0f-82c4-cb5a233c29ef","Type":"ContainerStarted","Data":"5c6e57b0d6c38d122b43f22d76bdbcf8c0bd8818b97ae5276a35c55cd9d3d120"}
Sep 10 12:38:18.715602 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:18.715287    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-bgjwx"
Sep 10 12:38:18.736983 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:18.736938    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-bgjwx" podStartSLOduration=2.034130595 podStartE2EDuration="4.7369244s" totalImagesPullingTime="2.702793805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:38:18.735937944 +0000 UTC m=+472.660849431" watchObservedRunningTime="2026-09-10 12:38:18.7369244 +0000 UTC m=+472.661835887"
Sep 10 12:38:20.712247 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:20.712217    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"]
Sep 10 12:38:20.715274 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:20.715258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:20.719413 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:20.719389    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v8mmc\""
Sep 10 12:38:20.730619 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:20.730591    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"]
Sep 10 12:38:20.790122 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:20.790087    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e70c6ca0-b835-4251-b614-7eb33e50e6c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" (UID: \"e70c6ca0-b835-4251-b614-7eb33e50e6c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:20.790122 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:20.790129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lst68\" (UniqueName: \"kubernetes.io/projected/e70c6ca0-b835-4251-b614-7eb33e50e6c4-kube-api-access-lst68\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" (UID: \"e70c6ca0-b835-4251-b614-7eb33e50e6c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:20.891623 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:20.891582    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e70c6ca0-b835-4251-b614-7eb33e50e6c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" (UID: \"e70c6ca0-b835-4251-b614-7eb33e50e6c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:20.891623 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:20.891627    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lst68\" (UniqueName: \"kubernetes.io/projected/e70c6ca0-b835-4251-b614-7eb33e50e6c4-kube-api-access-lst68\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" (UID: \"e70c6ca0-b835-4251-b614-7eb33e50e6c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:20.891987 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:20.891964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e70c6ca0-b835-4251-b614-7eb33e50e6c4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" (UID: \"e70c6ca0-b835-4251-b614-7eb33e50e6c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:20.906317 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:20.906291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lst68\" (UniqueName: \"kubernetes.io/projected/e70c6ca0-b835-4251-b614-7eb33e50e6c4-kube-api-access-lst68\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" (UID: \"e70c6ca0-b835-4251-b614-7eb33e50e6c4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:21.025711 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:21.025609    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:21.302933 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:21.302908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"]
Sep 10 12:38:21.305030 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:38:21.305000    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70c6ca0_b835_4251_b614_7eb33e50e6c4.slice/crio-4985884431cabb6038135372b20a5f97a0d08f05ba18c88e3eae1ad420a00967 WatchSource:0}: Error finding container 4985884431cabb6038135372b20a5f97a0d08f05ba18c88e3eae1ad420a00967: Status 404 returned error can't find the container with id 4985884431cabb6038135372b20a5f97a0d08f05ba18c88e3eae1ad420a00967
Sep 10 12:38:21.728614 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:21.728580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7" event={"ID":"e70c6ca0-b835-4251-b614-7eb33e50e6c4","Type":"ContainerStarted","Data":"4985884431cabb6038135372b20a5f97a0d08f05ba18c88e3eae1ad420a00967"}
Sep 10 12:38:23.729826 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:23.729786    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-clc6x"]
Sep 10 12:38:23.733446 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:23.733420    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-clc6x"
Sep 10 12:38:23.737376 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:23.737352    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-nkqrk\""
Sep 10 12:38:23.737639 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:23.737600    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 12:38:23.761080 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:23.761053    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-clc6x"]
Sep 10 12:38:23.818935 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:23.818876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbr2k\" (UniqueName: \"kubernetes.io/projected/5b880597-db64-45fa-bcdb-2ed1808548d1-kube-api-access-mbr2k\") pod \"dns-operator-controller-manager-7b84cd7c9c-clc6x\" (UID: \"5b880597-db64-45fa-bcdb-2ed1808548d1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-clc6x"
Sep 10 12:38:23.920406 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:23.920371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbr2k\" (UniqueName: \"kubernetes.io/projected/5b880597-db64-45fa-bcdb-2ed1808548d1-kube-api-access-mbr2k\") pod \"dns-operator-controller-manager-7b84cd7c9c-clc6x\" (UID: \"5b880597-db64-45fa-bcdb-2ed1808548d1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-clc6x"
Sep 10 12:38:23.932578 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:23.932537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbr2k\" (UniqueName: \"kubernetes.io/projected/5b880597-db64-45fa-bcdb-2ed1808548d1-kube-api-access-mbr2k\") pod \"dns-operator-controller-manager-7b84cd7c9c-clc6x\" (UID: \"5b880597-db64-45fa-bcdb-2ed1808548d1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-clc6x"
Sep 10 12:38:24.045242 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:24.045152    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-clc6x"
Sep 10 12:38:25.075120 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:25.074895    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-clc6x"]
Sep 10 12:38:25.077589 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:38:25.077541    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b880597_db64_45fa_bcdb_2ed1808548d1.slice/crio-2fb40f2c4b3934b50394654812a37c057d253c8545accdac00ac4f6e4551a16c WatchSource:0}: Error finding container 2fb40f2c4b3934b50394654812a37c057d253c8545accdac00ac4f6e4551a16c: Status 404 returned error can't find the container with id 2fb40f2c4b3934b50394654812a37c057d253c8545accdac00ac4f6e4551a16c
Sep 10 12:38:25.752962 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:25.752924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7" event={"ID":"e70c6ca0-b835-4251-b614-7eb33e50e6c4","Type":"ContainerStarted","Data":"913315f05a6a93ea2d7d04070852ad3538962e204eed0d77b4e113e05575d573"}
Sep 10 12:38:25.753145 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:25.753029    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:25.754477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:25.754450    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-clc6x" event={"ID":"5b880597-db64-45fa-bcdb-2ed1808548d1","Type":"ContainerStarted","Data":"2fb40f2c4b3934b50394654812a37c057d253c8545accdac00ac4f6e4551a16c"}
Sep 10 12:38:25.804827 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:25.804774    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7" podStartSLOduration=2.23286458 podStartE2EDuration="5.804756844s" totalImagesPullingTime="3.571892264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:38:25.801335386 +0000 UTC m=+479.726246874" watchObservedRunningTime="2026-09-10 12:38:25.804756844 +0000 UTC m=+479.729668332"
Sep 10 12:38:26.604444 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:26.604413    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"]
Sep 10 12:38:26.608372 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:26.608350    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:26.615611 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:26.615585    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-68p9c\""
Sep 10 12:38:26.632665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:26.632621    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"]
Sep 10 12:38:26.746119 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:26.746086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kc7f\" (UniqueName: \"kubernetes.io/projected/96a518b6-72ae-4a94-aec7-eae6cd36a9b0-kube-api-access-9kc7f\") pod \"limitador-operator-controller-manager-74495c69f8-grpbl\" (UID: \"96a518b6-72ae-4a94-aec7-eae6cd36a9b0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:26.803813 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:26.803790    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 12:38:26.847726 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:26.847697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kc7f\" (UniqueName: \"kubernetes.io/projected/96a518b6-72ae-4a94-aec7-eae6cd36a9b0-kube-api-access-9kc7f\") pod \"limitador-operator-controller-manager-74495c69f8-grpbl\" (UID: \"96a518b6-72ae-4a94-aec7-eae6cd36a9b0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:26.859458 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:26.859403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kc7f\" (UniqueName: \"kubernetes.io/projected/96a518b6-72ae-4a94-aec7-eae6cd36a9b0-kube-api-access-9kc7f\") pod \"limitador-operator-controller-manager-74495c69f8-grpbl\" (UID: \"96a518b6-72ae-4a94-aec7-eae6cd36a9b0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:26.921211 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:26.921169    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:27.201773 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.201734    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"]
Sep 10 12:38:27.204212 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:38:27.204188    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a518b6_72ae_4a94_aec7_eae6cd36a9b0.slice/crio-16ffff78fcc8b954e520350d8e10ffa9b42940ebbbb7366803cfb5c10df9739e WatchSource:0}: Error finding container 16ffff78fcc8b954e520350d8e10ffa9b42940ebbbb7366803cfb5c10df9739e: Status 404 returned error can't find the container with id 16ffff78fcc8b954e520350d8e10ffa9b42940ebbbb7366803cfb5c10df9739e
Sep 10 12:38:27.728223 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.728190    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"]
Sep 10 12:38:27.732633 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.732616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"
Sep 10 12:38:27.735276 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.735256    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 12:38:27.735792 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.735761    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 12:38:27.736111 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.736083    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nwphl\""
Sep 10 12:38:27.745580 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.745452    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"]
Sep 10 12:38:27.765498 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.765447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-clc6x" event={"ID":"5b880597-db64-45fa-bcdb-2ed1808548d1","Type":"ContainerStarted","Data":"165216ea96a54d4fcbe31208d8c6557b806cd20a53733e028c277b6651cb0be2"}
Sep 10 12:38:27.765671 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.765538    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-clc6x"
Sep 10 12:38:27.767004 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.766978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl" event={"ID":"96a518b6-72ae-4a94-aec7-eae6cd36a9b0","Type":"ContainerStarted","Data":"16ffff78fcc8b954e520350d8e10ffa9b42940ebbbb7366803cfb5c10df9739e"}
Sep 10 12:38:27.785076 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.785038    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-clc6x" podStartSLOduration=3.064639303 podStartE2EDuration="4.785022353s" totalImagesPullingTime="1.72038305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:38:27.783473333 +0000 UTC m=+481.708384822" watchObservedRunningTime="2026-09-10 12:38:27.785022353 +0000 UTC m=+481.709933842"
Sep 10 12:38:27.856343 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.856313    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6765bc4-ff39-4903-b62b-caae6d75bd72-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nww57\" (UID: \"b6765bc4-ff39-4903-b62b-caae6d75bd72\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"
Sep 10 12:38:27.856479 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.856428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfd6n\" (UniqueName: \"kubernetes.io/projected/b6765bc4-ff39-4903-b62b-caae6d75bd72-kube-api-access-vfd6n\") pod \"kuadrant-console-plugin-fbdd4f445-nww57\" (UID: \"b6765bc4-ff39-4903-b62b-caae6d75bd72\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"
Sep 10 12:38:27.856479 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.856473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6765bc4-ff39-4903-b62b-caae6d75bd72-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nww57\" (UID: \"b6765bc4-ff39-4903-b62b-caae6d75bd72\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"
Sep 10 12:38:27.957917 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.957884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfd6n\" (UniqueName: \"kubernetes.io/projected/b6765bc4-ff39-4903-b62b-caae6d75bd72-kube-api-access-vfd6n\") pod \"kuadrant-console-plugin-fbdd4f445-nww57\" (UID: \"b6765bc4-ff39-4903-b62b-caae6d75bd72\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"
Sep 10 12:38:27.958074 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.957928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6765bc4-ff39-4903-b62b-caae6d75bd72-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nww57\" (UID: \"b6765bc4-ff39-4903-b62b-caae6d75bd72\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"
Sep 10 12:38:27.958074 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.957982    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6765bc4-ff39-4903-b62b-caae6d75bd72-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nww57\" (UID: \"b6765bc4-ff39-4903-b62b-caae6d75bd72\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"
Sep 10 12:38:27.958151 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:27.958104    2563 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 10 12:38:27.958184 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:27.958167    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6765bc4-ff39-4903-b62b-caae6d75bd72-plugin-serving-cert podName:b6765bc4-ff39-4903-b62b-caae6d75bd72 nodeName:}" failed. No retries permitted until 2026-09-10 12:38:28.458148203 +0000 UTC m=+482.383059684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b6765bc4-ff39-4903-b62b-caae6d75bd72-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-nww57" (UID: "b6765bc4-ff39-4903-b62b-caae6d75bd72") : secret "plugin-serving-cert" not found
Sep 10 12:38:27.958586 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.958540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b6765bc4-ff39-4903-b62b-caae6d75bd72-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-nww57\" (UID: \"b6765bc4-ff39-4903-b62b-caae6d75bd72\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"
Sep 10 12:38:27.967514 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:27.967493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfd6n\" (UniqueName: \"kubernetes.io/projected/b6765bc4-ff39-4903-b62b-caae6d75bd72-kube-api-access-vfd6n\") pod \"kuadrant-console-plugin-fbdd4f445-nww57\" (UID: \"b6765bc4-ff39-4903-b62b-caae6d75bd72\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"
Sep 10 12:38:28.463141 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:28.463102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6765bc4-ff39-4903-b62b-caae6d75bd72-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nww57\" (UID: \"b6765bc4-ff39-4903-b62b-caae6d75bd72\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"
Sep 10 12:38:28.465489 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:28.465460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6765bc4-ff39-4903-b62b-caae6d75bd72-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-nww57\" (UID: \"b6765bc4-ff39-4903-b62b-caae6d75bd72\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"
Sep 10 12:38:28.645363 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:28.645324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"
Sep 10 12:38:28.919686 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:28.919657    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57"]
Sep 10 12:38:28.922603 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:38:28.922576    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6765bc4_ff39_4903_b62b_caae6d75bd72.slice/crio-cfcf7a08c287dd198030432112df251feddd43ad07031b182446c14008ab1a54 WatchSource:0}: Error finding container cfcf7a08c287dd198030432112df251feddd43ad07031b182446c14008ab1a54: Status 404 returned error can't find the container with id cfcf7a08c287dd198030432112df251feddd43ad07031b182446c14008ab1a54
Sep 10 12:38:29.721443 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:29.721414    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-bgjwx"
Sep 10 12:38:29.782155 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:29.782084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57" event={"ID":"b6765bc4-ff39-4903-b62b-caae6d75bd72","Type":"ContainerStarted","Data":"cfcf7a08c287dd198030432112df251feddd43ad07031b182446c14008ab1a54"}
Sep 10 12:38:30.790464 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:30.790373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl" event={"ID":"96a518b6-72ae-4a94-aec7-eae6cd36a9b0","Type":"ContainerStarted","Data":"c6a20049207afb5c2e571dc5592e037bfa64dc0d33e31c7376c7fda8023cb1ea"}
Sep 10 12:38:30.790901 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:30.790466    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:30.826434 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:30.826370    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl" podStartSLOduration=1.5969913070000001 podStartE2EDuration="4.826348576s" totalImagesPullingTime="3.229357269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:38:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:38:30.823769386 +0000 UTC m=+484.748680874" watchObservedRunningTime="2026-09-10 12:38:30.826348576 +0000 UTC m=+484.751260067"
Sep 10 12:38:33.808501 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:33.808464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57" event={"ID":"b6765bc4-ff39-4903-b62b-caae6d75bd72","Type":"ContainerStarted","Data":"6d91262a30f735d869f476d0bc008763b0bef15224c0df36bfb85cee948e2b06"}
Sep 10 12:38:33.845608 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:33.845544    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-nww57" podStartSLOduration=2.163739455 podStartE2EDuration="6.845529222s" totalImagesPullingTime="4.681789767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:38:33.842457099 +0000 UTC m=+487.767368597" watchObservedRunningTime="2026-09-10 12:38:33.845529222 +0000 UTC m=+487.770440710"
Sep 10 12:38:36.761601 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:36.761546    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:38.774654 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.774619    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-clc6x"
Sep 10 12:38:38.912103 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.912066    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"]
Sep 10 12:38:38.912291 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.912249    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" containerName="manager" containerID="cri-o://913315f05a6a93ea2d7d04070852ad3538962e204eed0d77b4e113e05575d573" gracePeriod=2
Sep 10 12:38:38.914762 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:38.914728    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:38.915041 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.915020    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"]
Sep 10 12:38:38.928967 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.928941    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"]
Sep 10 12:38:38.929184 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.929161    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" containerName="manager" containerID="cri-o://c6a20049207afb5c2e571dc5592e037bfa64dc0d33e31c7376c7fda8023cb1ea" gracePeriod=2
Sep 10 12:38:38.931182 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.931163    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:38.941994 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.941971    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"]
Sep 10 12:38:38.947454 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:38.947420    2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"96a518b6-72ae-4a94-aec7-eae6cd36a9b0\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-grpbl\": pods \"limitador-operator-controller-manager-74495c69f8-grpbl\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:38.952592 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:38.952544    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:38.955031 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.954980    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"]
Sep 10 12:38:38.955248 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:38.955220    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-grpbl\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:38.955325 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.955312    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" containerName="manager"
Sep 10 12:38:38.955374 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.955325    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" containerName="manager"
Sep 10 12:38:38.955374 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.955335    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" containerName="manager"
Sep 10 12:38:38.955374 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.955341    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" containerName="manager"
Sep 10 12:38:38.955479 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.955407    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" containerName="manager"
Sep 10 12:38:38.955479 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.955414    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" containerName="manager"
Sep 10 12:38:38.958171 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:38.958148    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:38.958357 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.958344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"
Sep 10 12:38:38.960206 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:38.960180    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:38.965077 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.965051    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-92vhl"]
Sep 10 12:38:38.969116 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.969097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-92vhl"
Sep 10 12:38:38.974787 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.974760    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"]
Sep 10 12:38:38.990456 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:38.990426    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-92vhl"]
Sep 10 12:38:38.997983 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:38.997954    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-grpbl\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:39.000079 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.000055    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:39.002182 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.002161    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-grpbl\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:39.055342 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.055081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvv8c\" (UniqueName: \"kubernetes.io/projected/cdc478b1-236b-4e43-9914-9826ce71ef44-kube-api-access-fvv8c\") pod \"limitador-operator-controller-manager-74495c69f8-92vhl\" (UID: \"cdc478b1-236b-4e43-9914-9826ce71ef44\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-92vhl"
Sep 10 12:38:39.055342 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.055131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23672568-5fbf-4797-bb90-8bcc65c63b90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qj6j6\" (UID: \"23672568-5fbf-4797-bb90-8bcc65c63b90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"
Sep 10 12:38:39.055342 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.055279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqmkn\" (UniqueName: \"kubernetes.io/projected/23672568-5fbf-4797-bb90-8bcc65c63b90-kube-api-access-pqmkn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qj6j6\" (UID: \"23672568-5fbf-4797-bb90-8bcc65c63b90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"
Sep 10 12:38:39.156727 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.156702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqmkn\" (UniqueName: \"kubernetes.io/projected/23672568-5fbf-4797-bb90-8bcc65c63b90-kube-api-access-pqmkn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qj6j6\" (UID: \"23672568-5fbf-4797-bb90-8bcc65c63b90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"
Sep 10 12:38:39.156820 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.156758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvv8c\" (UniqueName: \"kubernetes.io/projected/cdc478b1-236b-4e43-9914-9826ce71ef44-kube-api-access-fvv8c\") pod \"limitador-operator-controller-manager-74495c69f8-92vhl\" (UID: \"cdc478b1-236b-4e43-9914-9826ce71ef44\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-92vhl"
Sep 10 12:38:39.157004 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.156989    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23672568-5fbf-4797-bb90-8bcc65c63b90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qj6j6\" (UID: \"23672568-5fbf-4797-bb90-8bcc65c63b90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"
Sep 10 12:38:39.157331 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.157315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23672568-5fbf-4797-bb90-8bcc65c63b90-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qj6j6\" (UID: \"23672568-5fbf-4797-bb90-8bcc65c63b90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"
Sep 10 12:38:39.172021 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.171996    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvv8c\" (UniqueName: \"kubernetes.io/projected/cdc478b1-236b-4e43-9914-9826ce71ef44-kube-api-access-fvv8c\") pod \"limitador-operator-controller-manager-74495c69f8-92vhl\" (UID: \"cdc478b1-236b-4e43-9914-9826ce71ef44\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-92vhl"
Sep 10 12:38:39.173268 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.173244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqmkn\" (UniqueName: \"kubernetes.io/projected/23672568-5fbf-4797-bb90-8bcc65c63b90-kube-api-access-pqmkn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qj6j6\" (UID: \"23672568-5fbf-4797-bb90-8bcc65c63b90\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"
Sep 10 12:38:39.176017 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.175997    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:39.177993 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.177977    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:39.178185 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.178166    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:39.180943 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.180923    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-grpbl\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:39.183193 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.183173    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:39.185176 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.185145    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-grpbl\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:39.258120 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.258091    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lst68\" (UniqueName: \"kubernetes.io/projected/e70c6ca0-b835-4251-b614-7eb33e50e6c4-kube-api-access-lst68\") pod \"e70c6ca0-b835-4251-b614-7eb33e50e6c4\" (UID: \"e70c6ca0-b835-4251-b614-7eb33e50e6c4\") "
Sep 10 12:38:39.258255 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.258158    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kc7f\" (UniqueName: \"kubernetes.io/projected/96a518b6-72ae-4a94-aec7-eae6cd36a9b0-kube-api-access-9kc7f\") pod \"96a518b6-72ae-4a94-aec7-eae6cd36a9b0\" (UID: \"96a518b6-72ae-4a94-aec7-eae6cd36a9b0\") "
Sep 10 12:38:39.258255 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.258198    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e70c6ca0-b835-4251-b614-7eb33e50e6c4-extensions-socket-volume\") pod \"e70c6ca0-b835-4251-b614-7eb33e50e6c4\" (UID: \"e70c6ca0-b835-4251-b614-7eb33e50e6c4\") "
Sep 10 12:38:39.258656 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.258630    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70c6ca0-b835-4251-b614-7eb33e50e6c4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e70c6ca0-b835-4251-b614-7eb33e50e6c4" (UID: "e70c6ca0-b835-4251-b614-7eb33e50e6c4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:38:39.260236 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.260213    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70c6ca0-b835-4251-b614-7eb33e50e6c4-kube-api-access-lst68" (OuterVolumeSpecName: "kube-api-access-lst68") pod "e70c6ca0-b835-4251-b614-7eb33e50e6c4" (UID: "e70c6ca0-b835-4251-b614-7eb33e50e6c4"). InnerVolumeSpecName "kube-api-access-lst68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:38:39.260305 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.260257    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a518b6-72ae-4a94-aec7-eae6cd36a9b0-kube-api-access-9kc7f" (OuterVolumeSpecName: "kube-api-access-9kc7f") pod "96a518b6-72ae-4a94-aec7-eae6cd36a9b0" (UID: "96a518b6-72ae-4a94-aec7-eae6cd36a9b0"). InnerVolumeSpecName "kube-api-access-9kc7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:38:39.353178 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.353075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"
Sep 10 12:38:39.359379 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.359346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-92vhl"
Sep 10 12:38:39.359931 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.359905    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kc7f\" (UniqueName: \"kubernetes.io/projected/96a518b6-72ae-4a94-aec7-eae6cd36a9b0-kube-api-access-9kc7f\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:38:39.360023 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.359935    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e70c6ca0-b835-4251-b614-7eb33e50e6c4-extensions-socket-volume\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:38:39.360023 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.359949    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lst68\" (UniqueName: \"kubernetes.io/projected/e70c6ca0-b835-4251-b614-7eb33e50e6c4-kube-api-access-lst68\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:38:39.646491 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:38:39.646450    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc478b1_236b_4e43_9914_9826ce71ef44.slice/crio-398023fcaee5cb4258dcb9a6ef3dbafd70458cd57d3eadfea1e711cf47455198 WatchSource:0}: Error finding container 398023fcaee5cb4258dcb9a6ef3dbafd70458cd57d3eadfea1e711cf47455198: Status 404 returned error can't find the container with id 398023fcaee5cb4258dcb9a6ef3dbafd70458cd57d3eadfea1e711cf47455198
Sep 10 12:38:39.648690 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.648661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-92vhl"]
Sep 10 12:38:39.664188 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.664166    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"]
Sep 10 12:38:39.667460 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:38:39.667432    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23672568_5fbf_4797_bb90_8bcc65c63b90.slice/crio-3f07885220c8a7c2a298355b3a28baadd11f28817bb27091808549d3afdb66bc WatchSource:0}: Error finding container 3f07885220c8a7c2a298355b3a28baadd11f28817bb27091808549d3afdb66bc: Status 404 returned error can't find the container with id 3f07885220c8a7c2a298355b3a28baadd11f28817bb27091808549d3afdb66bc
Sep 10 12:38:39.845262 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.845227    2563 generic.go:356] "Generic (PLEG): container finished" podID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" containerID="913315f05a6a93ea2d7d04070852ad3538962e204eed0d77b4e113e05575d573" exitCode=0
Sep 10 12:38:39.845730 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.845297    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:39.845730 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.845333    2563 scope.go:117] "RemoveContainer" containerID="913315f05a6a93ea2d7d04070852ad3538962e204eed0d77b4e113e05575d573"
Sep 10 12:38:39.847502 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.847477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-92vhl" event={"ID":"cdc478b1-236b-4e43-9914-9826ce71ef44","Type":"ContainerStarted","Data":"1e48b1838bd699dcef4f76895fa6ed49af295165c28e9b89e3a441cb0aae26e8"}
Sep 10 12:38:39.847771 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.847737    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-92vhl" event={"ID":"cdc478b1-236b-4e43-9914-9826ce71ef44","Type":"ContainerStarted","Data":"398023fcaee5cb4258dcb9a6ef3dbafd70458cd57d3eadfea1e711cf47455198"}
Sep 10 12:38:39.847920 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.847905    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-92vhl"
Sep 10 12:38:39.849988 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.849961    2563 generic.go:356] "Generic (PLEG): container finished" podID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" containerID="c6a20049207afb5c2e571dc5592e037bfa64dc0d33e31c7376c7fda8023cb1ea" exitCode=0
Sep 10 12:38:39.850088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.850015    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:39.851032 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.851005    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:39.853480 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.853454    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6" event={"ID":"23672568-5fbf-4797-bb90-8bcc65c63b90","Type":"ContainerStarted","Data":"87c7e164ae0e7d5558ae9b445d2bfb57b209791eeb4b5e6b1cbf27350d863148"}
Sep 10 12:38:39.853588 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.853502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6" event={"ID":"23672568-5fbf-4797-bb90-8bcc65c63b90","Type":"ContainerStarted","Data":"3f07885220c8a7c2a298355b3a28baadd11f28817bb27091808549d3afdb66bc"}
Sep 10 12:38:39.853987 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.853965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"
Sep 10 12:38:39.854082 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.854023    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-grpbl\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:39.856883 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.856859    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:39.858181 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.858162    2563 scope.go:117] "RemoveContainer" containerID="913315f05a6a93ea2d7d04070852ad3538962e204eed0d77b4e113e05575d573"
Sep 10 12:38:39.858479 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.858458    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"913315f05a6a93ea2d7d04070852ad3538962e204eed0d77b4e113e05575d573\": container with ID starting with 913315f05a6a93ea2d7d04070852ad3538962e204eed0d77b4e113e05575d573 not found: ID does not exist" containerID="913315f05a6a93ea2d7d04070852ad3538962e204eed0d77b4e113e05575d573"
Sep 10 12:38:39.858583 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.858484    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"913315f05a6a93ea2d7d04070852ad3538962e204eed0d77b4e113e05575d573"} err="failed to get container status \"913315f05a6a93ea2d7d04070852ad3538962e204eed0d77b4e113e05575d573\": rpc error: code = NotFound desc = could not find container \"913315f05a6a93ea2d7d04070852ad3538962e204eed0d77b4e113e05575d573\": container with ID starting with 913315f05a6a93ea2d7d04070852ad3538962e204eed0d77b4e113e05575d573 not found: ID does not exist"
Sep 10 12:38:39.858583 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.858501    2563 scope.go:117] "RemoveContainer" containerID="c6a20049207afb5c2e571dc5592e037bfa64dc0d33e31c7376c7fda8023cb1ea"
Sep 10 12:38:39.866694 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.866675    2563 scope.go:117] "RemoveContainer" containerID="c6a20049207afb5c2e571dc5592e037bfa64dc0d33e31c7376c7fda8023cb1ea"
Sep 10 12:38:39.866974 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.866955    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6a20049207afb5c2e571dc5592e037bfa64dc0d33e31c7376c7fda8023cb1ea\": container with ID starting with c6a20049207afb5c2e571dc5592e037bfa64dc0d33e31c7376c7fda8023cb1ea not found: ID does not exist" containerID="c6a20049207afb5c2e571dc5592e037bfa64dc0d33e31c7376c7fda8023cb1ea"
Sep 10 12:38:39.867029 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.866981    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6a20049207afb5c2e571dc5592e037bfa64dc0d33e31c7376c7fda8023cb1ea"} err="failed to get container status \"c6a20049207afb5c2e571dc5592e037bfa64dc0d33e31c7376c7fda8023cb1ea\": rpc error: code = NotFound desc = could not find container \"c6a20049207afb5c2e571dc5592e037bfa64dc0d33e31c7376c7fda8023cb1ea\": container with ID starting with c6a20049207afb5c2e571dc5592e037bfa64dc0d33e31c7376c7fda8023cb1ea not found: ID does not exist"
Sep 10 12:38:39.887648 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.887602    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6" podStartSLOduration=1.887588884 podStartE2EDuration="1.887588884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:38:39.887309287 +0000 UTC m=+493.812220775" watchObservedRunningTime="2026-09-10 12:38:39.887588884 +0000 UTC m=+493.812500372"
Sep 10 12:38:39.889448 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.889410    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-grpbl\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:39.914511 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:39.914419    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-92vhl" podStartSLOduration=1.914402306 podStartE2EDuration="1.914402306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:38:39.912629966 +0000 UTC m=+493.837541453" watchObservedRunningTime="2026-09-10 12:38:39.914402306 +0000 UTC m=+493.839313793"
Sep 10 12:38:39.914991 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.914965    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:39.916665 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:39.916630    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-grpbl\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:40.343227 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:40.343182    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"]
Sep 10 12:38:40.348187 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:40.348160    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:38:40.350900 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:40.350865    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-grpbl\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-grpbl"
Sep 10 12:38:40.353646 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:40.353617    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mwmm7\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mwmm7"
Sep 10 12:38:40.369688 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:40.369663    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"]
Sep 10 12:38:40.469461 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:40.469421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce40549a-16c6-4438-995e-01f1f68546da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vjw4t\" (UID: \"ce40549a-16c6-4438-995e-01f1f68546da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:38:40.469634 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:40.469472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-249d5\" (UniqueName: \"kubernetes.io/projected/ce40549a-16c6-4438-995e-01f1f68546da-kube-api-access-249d5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vjw4t\" (UID: \"ce40549a-16c6-4438-995e-01f1f68546da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:38:40.570451 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:40.570412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-249d5\" (UniqueName: \"kubernetes.io/projected/ce40549a-16c6-4438-995e-01f1f68546da-kube-api-access-249d5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vjw4t\" (UID: \"ce40549a-16c6-4438-995e-01f1f68546da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:38:40.570654 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:40.570526    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce40549a-16c6-4438-995e-01f1f68546da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vjw4t\" (UID: \"ce40549a-16c6-4438-995e-01f1f68546da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:38:40.570946 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:40.570925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce40549a-16c6-4438-995e-01f1f68546da-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vjw4t\" (UID: \"ce40549a-16c6-4438-995e-01f1f68546da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:38:40.582541 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:40.582519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-249d5\" (UniqueName: \"kubernetes.io/projected/ce40549a-16c6-4438-995e-01f1f68546da-kube-api-access-249d5\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vjw4t\" (UID: \"ce40549a-16c6-4438-995e-01f1f68546da\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:38:40.659319 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:40.659247    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:38:40.705986 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:40.705960    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a518b6-72ae-4a94-aec7-eae6cd36a9b0" path="/var/lib/kubelet/pods/96a518b6-72ae-4a94-aec7-eae6cd36a9b0/volumes"
Sep 10 12:38:40.706724 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:40.706606    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e70c6ca0-b835-4251-b614-7eb33e50e6c4" path="/var/lib/kubelet/pods/e70c6ca0-b835-4251-b614-7eb33e50e6c4/volumes"
Sep 10 12:38:40.954949 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:40.954926    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"]
Sep 10 12:38:40.956663 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:38:40.956627    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce40549a_16c6_4438_995e_01f1f68546da.slice/crio-34cf904f619fad27da9b573054abf493b568f97fed578941f33234ab4a32536c WatchSource:0}: Error finding container 34cf904f619fad27da9b573054abf493b568f97fed578941f33234ab4a32536c: Status 404 returned error can't find the container with id 34cf904f619fad27da9b573054abf493b568f97fed578941f33234ab4a32536c
Sep 10 12:38:41.866933 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:41.866897    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t" event={"ID":"ce40549a-16c6-4438-995e-01f1f68546da","Type":"ContainerStarted","Data":"0abdb3fce5bb3fc57c07d5f8832ee13efb596310f7c7dc96322a02e0fa5e1900"}
Sep 10 12:38:41.867112 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:41.866954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t" event={"ID":"ce40549a-16c6-4438-995e-01f1f68546da","Type":"ContainerStarted","Data":"34cf904f619fad27da9b573054abf493b568f97fed578941f33234ab4a32536c"}
Sep 10 12:38:41.867160 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:41.867121    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:38:41.891786 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:41.891728    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t" podStartSLOduration=1.89171355 podStartE2EDuration="1.89171355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:38:41.890108546 +0000 UTC m=+495.815020035" watchObservedRunningTime="2026-09-10 12:38:41.89171355 +0000 UTC m=+495.816625037"
Sep 10 12:38:50.862536 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:50.862504    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-92vhl"
Sep 10 12:38:51.869091 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:51.869063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"
Sep 10 12:38:52.875764 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:52.875737    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:38:52.934970 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:52.934939    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"]
Sep 10 12:38:52.935140 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:52.935098    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6" podUID="23672568-5fbf-4797-bb90-8bcc65c63b90" containerName="manager" containerID="cri-o://87c7e164ae0e7d5558ae9b445d2bfb57b209791eeb4b5e6b1cbf27350d863148" gracePeriod=10
Sep 10 12:38:53.196504 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.196477    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"
Sep 10 12:38:53.274624 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.274595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqmkn\" (UniqueName: \"kubernetes.io/projected/23672568-5fbf-4797-bb90-8bcc65c63b90-kube-api-access-pqmkn\") pod \"23672568-5fbf-4797-bb90-8bcc65c63b90\" (UID: \"23672568-5fbf-4797-bb90-8bcc65c63b90\") "
Sep 10 12:38:53.274783 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.274653    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23672568-5fbf-4797-bb90-8bcc65c63b90-extensions-socket-volume\") pod \"23672568-5fbf-4797-bb90-8bcc65c63b90\" (UID: \"23672568-5fbf-4797-bb90-8bcc65c63b90\") "
Sep 10 12:38:53.274991 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.274969    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23672568-5fbf-4797-bb90-8bcc65c63b90-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "23672568-5fbf-4797-bb90-8bcc65c63b90" (UID: "23672568-5fbf-4797-bb90-8bcc65c63b90"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:38:53.276934 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.276906    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23672568-5fbf-4797-bb90-8bcc65c63b90-kube-api-access-pqmkn" (OuterVolumeSpecName: "kube-api-access-pqmkn") pod "23672568-5fbf-4797-bb90-8bcc65c63b90" (UID: "23672568-5fbf-4797-bb90-8bcc65c63b90"). InnerVolumeSpecName "kube-api-access-pqmkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:38:53.375817 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.375777    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqmkn\" (UniqueName: \"kubernetes.io/projected/23672568-5fbf-4797-bb90-8bcc65c63b90-kube-api-access-pqmkn\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:38:53.375817 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.375808    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/23672568-5fbf-4797-bb90-8bcc65c63b90-extensions-socket-volume\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:38:53.930495 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.930458    2563 generic.go:356] "Generic (PLEG): container finished" podID="23672568-5fbf-4797-bb90-8bcc65c63b90" containerID="87c7e164ae0e7d5558ae9b445d2bfb57b209791eeb4b5e6b1cbf27350d863148" exitCode=0
Sep 10 12:38:53.930495 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.930496    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"
Sep 10 12:38:53.930933 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.930535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6" event={"ID":"23672568-5fbf-4797-bb90-8bcc65c63b90","Type":"ContainerDied","Data":"87c7e164ae0e7d5558ae9b445d2bfb57b209791eeb4b5e6b1cbf27350d863148"}
Sep 10 12:38:53.930933 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.930592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6" event={"ID":"23672568-5fbf-4797-bb90-8bcc65c63b90","Type":"ContainerDied","Data":"3f07885220c8a7c2a298355b3a28baadd11f28817bb27091808549d3afdb66bc"}
Sep 10 12:38:53.930933 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.930614    2563 scope.go:117] "RemoveContainer" containerID="87c7e164ae0e7d5558ae9b445d2bfb57b209791eeb4b5e6b1cbf27350d863148"
Sep 10 12:38:53.939231 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.939057    2563 scope.go:117] "RemoveContainer" containerID="87c7e164ae0e7d5558ae9b445d2bfb57b209791eeb4b5e6b1cbf27350d863148"
Sep 10 12:38:53.939334 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:38:53.939316    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c7e164ae0e7d5558ae9b445d2bfb57b209791eeb4b5e6b1cbf27350d863148\": container with ID starting with 87c7e164ae0e7d5558ae9b445d2bfb57b209791eeb4b5e6b1cbf27350d863148 not found: ID does not exist" containerID="87c7e164ae0e7d5558ae9b445d2bfb57b209791eeb4b5e6b1cbf27350d863148"
Sep 10 12:38:53.939370 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.939342    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c7e164ae0e7d5558ae9b445d2bfb57b209791eeb4b5e6b1cbf27350d863148"} err="failed to get container status \"87c7e164ae0e7d5558ae9b445d2bfb57b209791eeb4b5e6b1cbf27350d863148\": rpc error: code = NotFound desc = could not find container \"87c7e164ae0e7d5558ae9b445d2bfb57b209791eeb4b5e6b1cbf27350d863148\": container with ID starting with 87c7e164ae0e7d5558ae9b445d2bfb57b209791eeb4b5e6b1cbf27350d863148 not found: ID does not exist"
Sep 10 12:38:53.958654 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.958633    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"]
Sep 10 12:38:53.960519 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:53.960500    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qj6j6"]
Sep 10 12:38:54.710841 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:38:54.710807    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23672568-5fbf-4797-bb90-8bcc65c63b90" path="/var/lib/kubelet/pods/23672568-5fbf-4797-bb90-8bcc65c63b90/volumes"
Sep 10 12:39:09.298206 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.298176    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"]
Sep 10 12:39:09.298700 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.298662    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23672568-5fbf-4797-bb90-8bcc65c63b90" containerName="manager"
Sep 10 12:39:09.298700 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.298682    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="23672568-5fbf-4797-bb90-8bcc65c63b90" containerName="manager"
Sep 10 12:39:09.298818 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.298804    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="23672568-5fbf-4797-bb90-8bcc65c63b90" containerName="manager"
Sep 10 12:39:09.308321 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.308302    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.310927 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.310882    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-srhs8\""
Sep 10 12:39:09.311847 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.310980    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 12:39:09.314108 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.314085    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"]
Sep 10 12:39:09.397287 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.397250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.397455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.397303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.397455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.397357    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.397455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.397408    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.397602 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.397466    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.397602 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.397494    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.397602 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.397533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.397698 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.397608    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mlv9\" (UniqueName: \"kubernetes.io/projected/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-kube-api-access-8mlv9\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.397698 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.397635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.498783 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.498745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.498997 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.498802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mlv9\" (UniqueName: \"kubernetes.io/projected/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-kube-api-access-8mlv9\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.498997 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.498831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.498997 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.498908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.498997 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.498939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.498997 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.498972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.499491 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.499305    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.499491 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.499330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.499491 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.499375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.499491 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.499334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.499491 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.499401    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.499491 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.499437    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.499860 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.499706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.499860 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.499752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.501234 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.501208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.501710 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.501689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.507905 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.507876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.508176 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.508157    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mlv9\" (UniqueName: \"kubernetes.io/projected/dca38e13-c602-4bf5-8f9f-845c37b7ab9d-kube-api-access-8mlv9\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tsqwk\" (UID: \"dca38e13-c602-4bf5-8f9f-845c37b7ab9d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.619800 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.619719    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:09.895489 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:09.895461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"]
Sep 10 12:39:09.897971 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:39:09.897942    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca38e13_c602_4bf5_8f9f_845c37b7ab9d.slice/crio-43f5d9948d52b64e1ba16142ffde1e7fdfd39622d0edc75f6fa0792a8e235737 WatchSource:0}: Error finding container 43f5d9948d52b64e1ba16142ffde1e7fdfd39622d0edc75f6fa0792a8e235737: Status 404 returned error can't find the container with id 43f5d9948d52b64e1ba16142ffde1e7fdfd39622d0edc75f6fa0792a8e235737
Sep 10 12:39:10.014864 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:10.014833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk" event={"ID":"dca38e13-c602-4bf5-8f9f-845c37b7ab9d","Type":"ContainerStarted","Data":"43f5d9948d52b64e1ba16142ffde1e7fdfd39622d0edc75f6fa0792a8e235737"}
Sep 10 12:39:12.711257 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:12.711213    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:39:12.711665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:12.711284    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:39:12.711665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:12.711316    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:39:13.034088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:13.033999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk" event={"ID":"dca38e13-c602-4bf5-8f9f-845c37b7ab9d","Type":"ContainerStarted","Data":"c9a71f97d3b2ec09df1b677292f3b2e572ae084b353b673e7859f8db7d7040eb"}
Sep 10 12:39:13.056098 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:13.056052    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk" podStartSLOduration=1.245009075 podStartE2EDuration="4.056036592s" totalImagesPullingTime="2.811027517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:39:13.055041318 +0000 UTC m=+526.979952819" watchObservedRunningTime="2026-09-10 12:39:13.056036592 +0000 UTC m=+526.980948080"
Sep 10 12:39:13.619940 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:13.619908    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:13.624494 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:13.624468    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:14.039107 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.039075    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:14.039976 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.039958    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:39:14.267782 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.267747    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:39:14.270402 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.270383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tm276"
Sep 10 12:39:14.273114 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.273094    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 12:39:14.283502 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.283477    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:39:14.297862 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.297799    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:39:14.343623 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.343595    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdxtn\" (UniqueName: \"kubernetes.io/projected/b61d0759-f7e6-4600-9f93-2b55f09edf54-kube-api-access-vdxtn\") pod \"limitador-limitador-84994f7ffc-tm276\" (UID: \"b61d0759-f7e6-4600-9f93-2b55f09edf54\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tm276"
Sep 10 12:39:14.343752 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.343681    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b61d0759-f7e6-4600-9f93-2b55f09edf54-config-file\") pod \"limitador-limitador-84994f7ffc-tm276\" (UID: \"b61d0759-f7e6-4600-9f93-2b55f09edf54\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tm276"
Sep 10 12:39:14.444927 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.444893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b61d0759-f7e6-4600-9f93-2b55f09edf54-config-file\") pod \"limitador-limitador-84994f7ffc-tm276\" (UID: \"b61d0759-f7e6-4600-9f93-2b55f09edf54\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tm276"
Sep 10 12:39:14.445093 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.444952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdxtn\" (UniqueName: \"kubernetes.io/projected/b61d0759-f7e6-4600-9f93-2b55f09edf54-kube-api-access-vdxtn\") pod \"limitador-limitador-84994f7ffc-tm276\" (UID: \"b61d0759-f7e6-4600-9f93-2b55f09edf54\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tm276"
Sep 10 12:39:14.445493 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.445475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b61d0759-f7e6-4600-9f93-2b55f09edf54-config-file\") pod \"limitador-limitador-84994f7ffc-tm276\" (UID: \"b61d0759-f7e6-4600-9f93-2b55f09edf54\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tm276"
Sep 10 12:39:14.454322 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.454294    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdxtn\" (UniqueName: \"kubernetes.io/projected/b61d0759-f7e6-4600-9f93-2b55f09edf54-kube-api-access-vdxtn\") pod \"limitador-limitador-84994f7ffc-tm276\" (UID: \"b61d0759-f7e6-4600-9f93-2b55f09edf54\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-tm276"
Sep 10 12:39:14.582114 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.582036    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-tm276"
Sep 10 12:39:14.855736 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:14.855711    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:39:14.856855 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:39:14.856814    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61d0759_f7e6_4600_9f93_2b55f09edf54.slice/crio-b56e32993a2620ce1574741ceea53712685bafe379ca86573a71c2d999cdc0c2 WatchSource:0}: Error finding container b56e32993a2620ce1574741ceea53712685bafe379ca86573a71c2d999cdc0c2: Status 404 returned error can't find the container with id b56e32993a2620ce1574741ceea53712685bafe379ca86573a71c2d999cdc0c2
Sep 10 12:39:15.045410 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:15.045380    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tm276" event={"ID":"b61d0759-f7e6-4600-9f93-2b55f09edf54","Type":"ContainerStarted","Data":"b56e32993a2620ce1574741ceea53712685bafe379ca86573a71c2d999cdc0c2"}
Sep 10 12:39:17.057856 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:17.057821    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-tm276" event={"ID":"b61d0759-f7e6-4600-9f93-2b55f09edf54","Type":"ContainerStarted","Data":"d938e05e82b3efeed8ec115910da1f31097e469d4b23dc823cd0b4a4c8d93b8e"}
Sep 10 12:39:17.058267 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:17.057921    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tm276"
Sep 10 12:39:17.093224 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:17.093176    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-tm276" podStartSLOduration=1.929717363 podStartE2EDuration="3.093159978s" totalImagesPullingTime="1.163442615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:39:17.091882667 +0000 UTC m=+531.016794157" watchObservedRunningTime="2026-09-10 12:39:17.093159978 +0000 UTC m=+531.018071466"
Sep 10 12:39:28.063732 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:39:28.063698    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-tm276"
Sep 10 12:40:29.380699 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:29.380661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:40:39.461119 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:39.461084    2563 generic.go:356] "Generic (PLEG): container finished" podID="dca38e13-c602-4bf5-8f9f-845c37b7ab9d" containerID="c9a71f97d3b2ec09df1b677292f3b2e572ae084b353b673e7859f8db7d7040eb" exitCode=0
Sep 10 12:40:39.461661 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:39.461162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk" event={"ID":"dca38e13-c602-4bf5-8f9f-845c37b7ab9d","Type":"ContainerDied","Data":"c9a71f97d3b2ec09df1b677292f3b2e572ae084b353b673e7859f8db7d7040eb"}
Sep 10 12:40:39.461734 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:39.461684    2563 scope.go:117] "RemoveContainer" containerID="c9a71f97d3b2ec09df1b677292f3b2e572ae084b353b673e7859f8db7d7040eb"
Sep 10 12:40:39.462686 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:39.462658    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:40:39.462777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:39.462721    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:40:39.462777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:39.462752    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:40:40.475810 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:40.475774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk" event={"ID":"dca38e13-c602-4bf5-8f9f-845c37b7ab9d","Type":"ContainerStarted","Data":"5cd8bdbfb3b3d08186e6c0999d03496dca7e8ec409651a02db57723f972770b6"}
Sep 10 12:40:40.620352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:40.620320    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:40:40.621663 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:40.621628    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tsqwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 10 12:40:40.621792 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:40.621689    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk" podUID="dca38e13-c602-4bf5-8f9f-845c37b7ab9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 10 12:40:41.620571 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:41.620526    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tsqwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 10 12:40:41.620956 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:41.620605    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk" podUID="dca38e13-c602-4bf5-8f9f-845c37b7ab9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 10 12:40:42.620421 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:42.620381    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tsqwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 10 12:40:42.620624 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:42.620444    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk" podUID="dca38e13-c602-4bf5-8f9f-845c37b7ab9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 10 12:40:43.620073 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:43.620036    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tsqwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 10 12:40:43.620247 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:43.620096    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk" podUID="dca38e13-c602-4bf5-8f9f-845c37b7ab9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 10 12:40:44.039268 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:44.039228    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:40:44.621112 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:44.621071    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tsqwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 10 12:40:44.621283 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:44.621134    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk" podUID="dca38e13-c602-4bf5-8f9f-845c37b7ab9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 10 12:40:45.620533 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:45.620499    2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tsqwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 10 12:40:45.620953 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:45.620582    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk" podUID="dca38e13-c602-4bf5-8f9f-845c37b7ab9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 10 12:40:46.624300 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:46.624265    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:40:46.625384 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:40:46.625364    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tsqwk"
Sep 10 12:41:05.737387 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:05.737349    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:41:11.305500 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:11.305468    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:41:16.706498 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.706463    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"]
Sep 10 12:41:16.709521 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.709501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.712894 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.712861    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 12:41:16.713023 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.712943    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 12:41:16.713209 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.713189    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-j76fb\""
Sep 10 12:41:16.713374 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.713309    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 12:41:16.715604 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.715585    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"]
Sep 10 12:41:16.840098 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.840051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44e48684-dba7-494f-9bc0-378b86c8bb4b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.840295 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.840117    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqph8\" (UniqueName: \"kubernetes.io/projected/44e48684-dba7-494f-9bc0-378b86c8bb4b-kube-api-access-xqph8\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.840295 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.840147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44e48684-dba7-494f-9bc0-378b86c8bb4b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.840295 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.840200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44e48684-dba7-494f-9bc0-378b86c8bb4b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.840412 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.840329    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44e48684-dba7-494f-9bc0-378b86c8bb4b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.840412 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.840366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44e48684-dba7-494f-9bc0-378b86c8bb4b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.941701 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.941661    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44e48684-dba7-494f-9bc0-378b86c8bb4b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.941701 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.941698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqph8\" (UniqueName: \"kubernetes.io/projected/44e48684-dba7-494f-9bc0-378b86c8bb4b-kube-api-access-xqph8\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.941945 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.941723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44e48684-dba7-494f-9bc0-378b86c8bb4b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.941945 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.941752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44e48684-dba7-494f-9bc0-378b86c8bb4b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.941945 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.941814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44e48684-dba7-494f-9bc0-378b86c8bb4b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.941945 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.941840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44e48684-dba7-494f-9bc0-378b86c8bb4b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.942146 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.942112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44e48684-dba7-494f-9bc0-378b86c8bb4b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.942189 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.942147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44e48684-dba7-494f-9bc0-378b86c8bb4b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.942189 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.942173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44e48684-dba7-494f-9bc0-378b86c8bb4b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.944002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.943973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44e48684-dba7-494f-9bc0-378b86c8bb4b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.944166 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.944150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44e48684-dba7-494f-9bc0-378b86c8bb4b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:16.951413 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:16.951384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqph8\" (UniqueName: \"kubernetes.io/projected/44e48684-dba7-494f-9bc0-378b86c8bb4b-kube-api-access-xqph8\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55\" (UID: \"44e48684-dba7-494f-9bc0-378b86c8bb4b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:17.022542 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:17.022454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:17.298474 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:17.298449    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"]
Sep 10 12:41:17.300487 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:41:17.300458    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e48684_dba7_494f_9bc0_378b86c8bb4b.slice/crio-0eedd9c2a75ee6b2ce53cbd26ce871022589e5fdb9bdb92d8837850a687fba70 WatchSource:0}: Error finding container 0eedd9c2a75ee6b2ce53cbd26ce871022589e5fdb9bdb92d8837850a687fba70: Status 404 returned error can't find the container with id 0eedd9c2a75ee6b2ce53cbd26ce871022589e5fdb9bdb92d8837850a687fba70
Sep 10 12:41:17.301968 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:17.301953    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:41:17.658790 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:17.658701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55" event={"ID":"44e48684-dba7-494f-9bc0-378b86c8bb4b","Type":"ContainerStarted","Data":"0eedd9c2a75ee6b2ce53cbd26ce871022589e5fdb9bdb92d8837850a687fba70"}
Sep 10 12:41:17.713207 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:17.713170    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:41:18.665003 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:18.664922    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55" event={"ID":"44e48684-dba7-494f-9bc0-378b86c8bb4b","Type":"ContainerStarted","Data":"8802f288bb623bc3e07d6342bfabb629b49dbd24b03cda098e5af2c7d8c1a80c"}
Sep 10 12:41:27.023277 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:27.023239    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:27.023891 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:27.023315    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:27.035776 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:27.035749    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:27.076712 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:27.076660    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55" podStartSLOduration=10.061387076 podStartE2EDuration="11.076643593s" totalImagesPullingTime="1.015256517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:41:18.685208874 +0000 UTC m=+652.610120362" watchObservedRunningTime="2026-09-10 12:41:27.076643593 +0000 UTC m=+661.001555081"
Sep 10 12:41:27.720392 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:27.720365    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55"
Sep 10 12:41:28.015684 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:28.015595    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:41:46.205614 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:41:46.205575    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:42:01.114879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.114839    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"]
Sep 10 12:42:01.119162 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.119141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.121705 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.121682    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 12:42:01.134178 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.134147    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"]
Sep 10 12:42:01.249750 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.249705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a9a835-70a1-48f1-bbed-8c7847263caa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.249938 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.249760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a9a835-70a1-48f1-bbed-8c7847263caa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.249938 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.249796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24klc\" (UniqueName: \"kubernetes.io/projected/c4a9a835-70a1-48f1-bbed-8c7847263caa-kube-api-access-24klc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.249938 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.249853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a9a835-70a1-48f1-bbed-8c7847263caa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.249938 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.249899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a9a835-70a1-48f1-bbed-8c7847263caa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.250222 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.249952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a9a835-70a1-48f1-bbed-8c7847263caa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.351107 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.351059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a9a835-70a1-48f1-bbed-8c7847263caa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.351107 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.351106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a9a835-70a1-48f1-bbed-8c7847263caa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.351355 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.351137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24klc\" (UniqueName: \"kubernetes.io/projected/c4a9a835-70a1-48f1-bbed-8c7847263caa-kube-api-access-24klc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.351355 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.351181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a9a835-70a1-48f1-bbed-8c7847263caa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.351355 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.351212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a9a835-70a1-48f1-bbed-8c7847263caa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.351355 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.351242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a9a835-70a1-48f1-bbed-8c7847263caa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.351683 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.351653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4a9a835-70a1-48f1-bbed-8c7847263caa-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.351765 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.351704    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4a9a835-70a1-48f1-bbed-8c7847263caa-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.351824 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.351806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4a9a835-70a1-48f1-bbed-8c7847263caa-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.353851 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.353825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4a9a835-70a1-48f1-bbed-8c7847263caa-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.354200 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.354182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4a9a835-70a1-48f1-bbed-8c7847263caa-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.359477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.359451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24klc\" (UniqueName: \"kubernetes.io/projected/c4a9a835-70a1-48f1-bbed-8c7847263caa-kube-api-access-24klc\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5gtcm\" (UID: \"c4a9a835-70a1-48f1-bbed-8c7847263caa\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.429734 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.429694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:01.705739 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.705714    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"]
Sep 10 12:42:01.707294 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:42:01.707266    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a9a835_70a1_48f1_bbed_8c7847263caa.slice/crio-b50d04464083acace17b095fd571707052a8e349cbdf8e789560eedf49054735 WatchSource:0}: Error finding container b50d04464083acace17b095fd571707052a8e349cbdf8e789560eedf49054735: Status 404 returned error can't find the container with id b50d04464083acace17b095fd571707052a8e349cbdf8e789560eedf49054735
Sep 10 12:42:01.888834 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:01.888793    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm" event={"ID":"c4a9a835-70a1-48f1-bbed-8c7847263caa","Type":"ContainerStarted","Data":"b50d04464083acace17b095fd571707052a8e349cbdf8e789560eedf49054735"}
Sep 10 12:42:02.895042 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:02.895010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm" event={"ID":"c4a9a835-70a1-48f1-bbed-8c7847263caa","Type":"ContainerStarted","Data":"65e3949e1083c677502bf37e26055bff30f8fcd7428ccb0d1bab50aa62a269cd"}
Sep 10 12:42:02.921537 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:02.921484    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm" podStartSLOduration=1.7143874590000001 podStartE2EDuration="1.921469717s" totalImagesPullingTime="207.082258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:42:02.920124283 +0000 UTC m=+696.845035771" watchObservedRunningTime="2026-09-10 12:42:02.921469717 +0000 UTC m=+696.846381203"
Sep 10 12:42:02.926624 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:02.926599    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:42:07.111483 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.111445    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"]
Sep 10 12:42:07.115859 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.115832    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.118377 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.118341    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 12:42:07.125059 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.125022    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"]
Sep 10 12:42:07.207044 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.207008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.207212 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.207055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.207212 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.207120    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.207212 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.207164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.207319 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.207212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.207319 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.207245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l7ws\" (UniqueName: \"kubernetes.io/projected/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-kube-api-access-5l7ws\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.307978 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.307936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l7ws\" (UniqueName: \"kubernetes.io/projected/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-kube-api-access-5l7ws\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.308166 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.307994    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.308166 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.308017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.308166 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.308042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.308166 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.308070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.308166 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.308107    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.308450 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.308428    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.308541 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.308519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.308616 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.308546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.310399 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.310375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.310594 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.310578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.316178 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.316156    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l7ws\" (UniqueName: \"kubernetes.io/projected/53652eb5-0e3a-47a3-ae74-eeff17a0dcdc-kube-api-access-5l7ws\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks\" (UID: \"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.428928 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.428899    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:07.705293 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.705267    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"]
Sep 10 12:42:07.707070 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:42:07.707043    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53652eb5_0e3a_47a3_ae74_eeff17a0dcdc.slice/crio-9268ec1ae03ea92c10831c89b61556fb5273cbe98afbb25bb6d1ca3a767a3001 WatchSource:0}: Error finding container 9268ec1ae03ea92c10831c89b61556fb5273cbe98afbb25bb6d1ca3a767a3001: Status 404 returned error can't find the container with id 9268ec1ae03ea92c10831c89b61556fb5273cbe98afbb25bb6d1ca3a767a3001
Sep 10 12:42:07.919139 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:07.919109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks" event={"ID":"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc","Type":"ContainerStarted","Data":"9268ec1ae03ea92c10831c89b61556fb5273cbe98afbb25bb6d1ca3a767a3001"}
Sep 10 12:42:08.804864 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:08.804829    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:42:08.925815 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:08.925783    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks" event={"ID":"53652eb5-0e3a-47a3-ae74-eeff17a0dcdc","Type":"ContainerStarted","Data":"3de9484d3c517b8504ea5257e3a612463f406dcebaf27e079551979cc4d3d258"}
Sep 10 12:42:08.943979 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:08.943920    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks" podStartSLOduration=1.750600945 podStartE2EDuration="1.943900145s" totalImagesPullingTime="193.2992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:42:08.94367194 +0000 UTC m=+702.868583430" watchObservedRunningTime="2026-09-10 12:42:08.943900145 +0000 UTC m=+702.868811633"
Sep 10 12:42:11.430660 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:11.430621    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:11.431036 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:11.430779    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:11.443191 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:11.443169    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:11.953448 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:11.953419    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5gtcm"
Sep 10 12:42:17.429640 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:17.429585    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:17.430201 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:17.429862    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:17.442398 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:17.442371    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:42:17.983191 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:42:17.983160    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks"
Sep 10 12:43:53.887880 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:53.887792    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"]
Sep 10 12:43:53.891422 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:53.891400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:53.896069 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:43:53.896034    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:53.896069 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:43:53.896051    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"e2e-worker-w0-30919a-openshift-default-dockercfg-sjwkd\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-30919a-openshift-default-dockercfg-sjwkd\"" type="*v1.Secret"
Sep 10 12:43:53.999619 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:53.999583    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"]
Sep 10 12:43:54.026043 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.026002    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-credential-socket\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.026252 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.026226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-token\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.026336 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.026279    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-podinfo\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.026336 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.026325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-workload-certs\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.026445 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.026410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-data\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.026506 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.026445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxw9f\" (UniqueName: \"kubernetes.io/projected/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-kube-api-access-cxw9f\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.026506 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.026473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istiod-ca-cert\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.026609 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.026570    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-workload-socket\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.026663 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.026648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-envoy\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.128123 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.128076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-data\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.128123 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.128127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxw9f\" (UniqueName: \"kubernetes.io/projected/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-kube-api-access-cxw9f\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.128338 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.128154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istiod-ca-cert\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.128338 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.128189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-workload-socket\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.128338 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.128260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-envoy\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.128338 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.128315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-credential-socket\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.128546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.128356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-token\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.128546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.128387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-podinfo\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.128546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.128430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-workload-certs\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.128546 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.128532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-data\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.128790 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.128756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-workload-certs\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.128973 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.128943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istiod-ca-cert\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.129093 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.128975    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-credential-socket\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.129152 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.129086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-workload-socket\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.130824 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.130801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-envoy\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.131279 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.131260    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-podinfo\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.152728 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.152651    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-token\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:54.153609 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:54.153588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxw9f\" (UniqueName: \"kubernetes.io/projected/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-kube-api-access-cxw9f\") pod \"e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") " pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:55.044464 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:55.044428    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-30919a-openshift-default-dockercfg-sjwkd\""
Sep 10 12:43:55.053632 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:55.053591    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:55.334208 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:43:55.334176    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf085c5f2_b9da_47e9_9342_7bd37db0b1f9.slice/crio-be00ccc6aa11aebebba22d369a48359f117cff4f72e47231724ecddb3852c637 WatchSource:0}: Error finding container be00ccc6aa11aebebba22d369a48359f117cff4f72e47231724ecddb3852c637: Status 404 returned error can't find the container with id be00ccc6aa11aebebba22d369a48359f117cff4f72e47231724ecddb3852c637
Sep 10 12:43:55.335491 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:55.335464    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"]
Sep 10 12:43:55.336466 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:55.336431    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:43:55.336523 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:55.336499    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:43:55.336580 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:55.336536    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:43:55.484266 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:55.484228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" event={"ID":"f085c5f2-b9da-47e9-9342-7bd37db0b1f9","Type":"ContainerStarted","Data":"8ee4255861bd0688cdbd3f46ee1af8a44b7ef62574466fb9f10072d21f507857"}
Sep 10 12:43:55.484447 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:55.484295    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" event={"ID":"f085c5f2-b9da-47e9-9342-7bd37db0b1f9","Type":"ContainerStarted","Data":"be00ccc6aa11aebebba22d369a48359f117cff4f72e47231724ecddb3852c637"}
Sep 10 12:43:55.529567 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:55.529492    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" podStartSLOduration=2.529472413 podStartE2EDuration="2.529472413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:43:55.529149846 +0000 UTC m=+809.454061346" watchObservedRunningTime="2026-09-10 12:43:55.529472413 +0000 UTC m=+809.454383900"
Sep 10 12:43:56.054334 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:56.054294    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:56.059545 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:56.059518    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:56.490140 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:56.490107    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:56.491116 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:56.491093    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:43:57.479212 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:43:57.479169    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:44:03.165180 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.165141    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"]
Sep 10 12:44:03.169030 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.169010    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.174171 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.174148    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-5abff0-openshift-default-dockercfg-h7rsh\""
Sep 10 12:44:03.192655 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.192630    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"]
Sep 10 12:44:03.322734 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.322694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-podinfo\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.322947 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.322763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-workload-socket\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.322947 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.322858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-data\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.322947 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.322898    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfmw9\" (UniqueName: \"kubernetes.io/projected/91e9c312-8b60-41ce-8bfe-5307b08a77c3-kube-api-access-jfmw9\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.323107 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.322983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-credential-socket\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.323107 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.323003    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-workload-certs\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.323107 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.323034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-token\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.323256 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.323125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istiod-ca-cert\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.323256 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.323151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-envoy\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.424202 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.424172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-workload-socket\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.424398 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.424220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-data\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.424398 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.424242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfmw9\" (UniqueName: \"kubernetes.io/projected/91e9c312-8b60-41ce-8bfe-5307b08a77c3-kube-api-access-jfmw9\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.424398 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.424279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-credential-socket\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.424398 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.424296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-workload-certs\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.424398 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.424327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-token\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.424398 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.424383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istiod-ca-cert\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.424737 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.424406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-envoy\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.424737 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.424439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-podinfo\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.424737 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.424607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-workload-socket\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.424737 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.424615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-credential-socket\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.424737 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.424677    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-data\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.424946 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.424747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-workload-certs\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.425291 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.425269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istiod-ca-cert\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.426807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.426784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-envoy\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.426929 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.426919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-podinfo\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.439882 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.439857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfmw9\" (UniqueName: \"kubernetes.io/projected/91e9c312-8b60-41ce-8bfe-5307b08a77c3-kube-api-access-jfmw9\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.442713 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.442681    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-token\") pod \"e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") " pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.479083 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.479058    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:03.773971 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:44:03.773944    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e9c312_8b60_41ce_8bfe_5307b08a77c3.slice/crio-a91c4711d976884173616425a0601508339cc05a830878ff2034fd669b29a5fc WatchSource:0}: Error finding container a91c4711d976884173616425a0601508339cc05a830878ff2034fd669b29a5fc: Status 404 returned error can't find the container with id a91c4711d976884173616425a0601508339cc05a830878ff2034fd669b29a5fc
Sep 10 12:44:03.774695 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.774673    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"]
Sep 10 12:44:03.776573 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.776520    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:03.776664 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.776614    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:03.776664 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:03.776652    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:04.536019 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:04.535980    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" event={"ID":"91e9c312-8b60-41ce-8bfe-5307b08a77c3","Type":"ContainerStarted","Data":"f89fce842d8ec0d4aa82c7d8ae0c54d942f006b8a822febbfde885cd25aeb97f"}
Sep 10 12:44:04.536500 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:04.536033    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" event={"ID":"91e9c312-8b60-41ce-8bfe-5307b08a77c3","Type":"ContainerStarted","Data":"a91c4711d976884173616425a0601508339cc05a830878ff2034fd669b29a5fc"}
Sep 10 12:44:04.656800 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:04.656745    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" podStartSLOduration=1.6567271319999999 podStartE2EDuration="1.656727132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:04.655809526 +0000 UTC m=+818.580721014" watchObservedRunningTime="2026-09-10 12:44:04.656727132 +0000 UTC m=+818.581638714"
Sep 10 12:44:05.480077 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:05.480044    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:05.485487 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:05.485458    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:05.541813 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:05.541783    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:05.542807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:05.542787    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:10.443994 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.443963    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"]
Sep 10 12:44:10.447634 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.447611    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.455665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.455639    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c8b03106-openshift-default-dockercfg-sft6x\""
Sep 10 12:44:10.491118 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.491092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3667f65-20e0-4574-9440-89862bfe9d5c-istiod-ca-cert\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.491257 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.491135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-workload-certs\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.491257 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.491156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-data\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.491257 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.491235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-credential-socket\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.491475 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.491285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-workload-socket\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.491475 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.491318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-podinfo\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.491475 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.491345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-envoy\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.491475 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.491368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jgc6\" (UniqueName: \"kubernetes.io/projected/b3667f65-20e0-4574-9440-89862bfe9d5c-kube-api-access-6jgc6\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.491475 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.491461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-token\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.541893 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.541858    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"]
Sep 10 12:44:10.592415 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.592371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3667f65-20e0-4574-9440-89862bfe9d5c-istiod-ca-cert\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.592603 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.592426    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-workload-certs\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.592603 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.592542    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-data\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.592744 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.592602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-credential-socket\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.592744 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.592642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-workload-socket\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.592744 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.592675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-podinfo\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.592744 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.592719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-envoy\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.592949 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.592759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jgc6\" (UniqueName: \"kubernetes.io/projected/b3667f65-20e0-4574-9440-89862bfe9d5c-kube-api-access-6jgc6\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.592949 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.592836    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-token\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.592949 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.592843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-workload-certs\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.592949 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.592915    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-data\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.593440 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.593414    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-workload-socket\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.593651 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.593448    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-credential-socket\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.593769 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.593658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3667f65-20e0-4574-9440-89862bfe9d5c-istiod-ca-cert\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.595362 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.595334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-envoy\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.595619 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.595599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-podinfo\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.614483 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.614449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jgc6\" (UniqueName: \"kubernetes.io/projected/b3667f65-20e0-4574-9440-89862bfe9d5c-kube-api-access-6jgc6\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.616186 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.616159    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-token\") pod \"e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") " pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:10.761530 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:10.761449    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:11.090408 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:11.090384    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"]
Sep 10 12:44:11.092346 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:44:11.092322    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3667f65_20e0_4574_9440_89862bfe9d5c.slice/crio-b7a9bbd93cba89f86dd4b259025c0f3f64d105bd9113819ce0bf3f2d126e3f5d WatchSource:0}: Error finding container b7a9bbd93cba89f86dd4b259025c0f3f64d105bd9113819ce0bf3f2d126e3f5d: Status 404 returned error can't find the container with id b7a9bbd93cba89f86dd4b259025c0f3f64d105bd9113819ce0bf3f2d126e3f5d
Sep 10 12:44:11.094724 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:11.094687    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:11.094831 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:11.094772    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:11.094831 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:11.094818    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:11.576075 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:11.576035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" event={"ID":"b3667f65-20e0-4574-9440-89862bfe9d5c","Type":"ContainerStarted","Data":"b7fcfd079e16fcca5e3a91a47ce18e861e9ceadbda05d6f1c493c6931ed4ce09"}
Sep 10 12:44:11.576459 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:11.576101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" event={"ID":"b3667f65-20e0-4574-9440-89862bfe9d5c","Type":"ContainerStarted","Data":"b7a9bbd93cba89f86dd4b259025c0f3f64d105bd9113819ce0bf3f2d126e3f5d"}
Sep 10 12:44:11.624719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:11.624670    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" podStartSLOduration=1.624654351 podStartE2EDuration="1.624654351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:11.622833482 +0000 UTC m=+825.547744974" watchObservedRunningTime="2026-09-10 12:44:11.624654351 +0000 UTC m=+825.549565839"
Sep 10 12:44:11.762518 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:11.762484    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:12.766613 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:12.766580    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:13.587266 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:13.587231    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:13.588376 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:13.588353    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:15.388967 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:15.388925    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:44:22.130702 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.130661    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"]
Sep 10 12:44:22.134117 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.134098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"
Sep 10 12:44:22.141093 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.141062    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c8b03106\""
Sep 10 12:44:22.141500 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.141477    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c8b03106-dockercfg-8zsw2\""
Sep 10 12:44:22.168855 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.168790    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"]
Sep 10 12:44:22.208957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.208922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffstb\" (UniqueName: \"kubernetes.io/projected/b714fed1-f0de-45b1-90a9-344d9268ad10-kube-api-access-ffstb\") pod \"payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4\" (UID: \"b714fed1-f0de-45b1-90a9-344d9268ad10\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"
Sep 10 12:44:22.209144 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.208983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b714fed1-f0de-45b1-90a9-344d9268ad10-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4\" (UID: \"b714fed1-f0de-45b1-90a9-344d9268ad10\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"
Sep 10 12:44:22.291012 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.290975    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"]
Sep 10 12:44:22.294828 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.294794    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"
Sep 10 12:44:22.309860 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.309835    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffstb\" (UniqueName: \"kubernetes.io/projected/b714fed1-f0de-45b1-90a9-344d9268ad10-kube-api-access-ffstb\") pod \"payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4\" (UID: \"b714fed1-f0de-45b1-90a9-344d9268ad10\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"
Sep 10 12:44:22.309980 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.309876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b714fed1-f0de-45b1-90a9-344d9268ad10-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4\" (UID: \"b714fed1-f0de-45b1-90a9-344d9268ad10\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"
Sep 10 12:44:22.310566 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.310529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b714fed1-f0de-45b1-90a9-344d9268ad10-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4\" (UID: \"b714fed1-f0de-45b1-90a9-344d9268ad10\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"
Sep 10 12:44:22.362946 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.362910    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"]
Sep 10 12:44:22.363900 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.363863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffstb\" (UniqueName: \"kubernetes.io/projected/b714fed1-f0de-45b1-90a9-344d9268ad10-kube-api-access-ffstb\") pod \"payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4\" (UID: \"b714fed1-f0de-45b1-90a9-344d9268ad10\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"
Sep 10 12:44:22.410984 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.410893    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbp6f\" (UniqueName: \"kubernetes.io/projected/7758f874-e331-46b3-8671-551a8ff7d8ad-kube-api-access-gbp6f\") pod \"payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj\" (UID: \"7758f874-e331-46b3-8671-551a8ff7d8ad\") " pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"
Sep 10 12:44:22.411120 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.411016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7758f874-e331-46b3-8671-551a8ff7d8ad-plugins-config-volume\") pod \"payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj\" (UID: \"7758f874-e331-46b3-8671-551a8ff7d8ad\") " pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"
Sep 10 12:44:22.444851 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.444821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"
Sep 10 12:44:22.512479 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.512424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbp6f\" (UniqueName: \"kubernetes.io/projected/7758f874-e331-46b3-8671-551a8ff7d8ad-kube-api-access-gbp6f\") pod \"payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj\" (UID: \"7758f874-e331-46b3-8671-551a8ff7d8ad\") " pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"
Sep 10 12:44:22.512677 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.512538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7758f874-e331-46b3-8671-551a8ff7d8ad-plugins-config-volume\") pod \"payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj\" (UID: \"7758f874-e331-46b3-8671-551a8ff7d8ad\") " pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"
Sep 10 12:44:22.513603 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.513521    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7758f874-e331-46b3-8671-551a8ff7d8ad-plugins-config-volume\") pod \"payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj\" (UID: \"7758f874-e331-46b3-8671-551a8ff7d8ad\") " pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"
Sep 10 12:44:22.526770 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.526738    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbp6f\" (UniqueName: \"kubernetes.io/projected/7758f874-e331-46b3-8671-551a8ff7d8ad-kube-api-access-gbp6f\") pod \"payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj\" (UID: \"7758f874-e331-46b3-8671-551a8ff7d8ad\") " pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"
Sep 10 12:44:22.605566 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.605524    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"
Sep 10 12:44:22.770312 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.770280    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"]
Sep 10 12:44:22.771497 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:44:22.771456    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb714fed1_f0de_45b1_90a9_344d9268ad10.slice/crio-bf2c34fb184a8e684b64b0e4cf930da8ecf1a405c3cf532400477c081863d38b WatchSource:0}: Error finding container bf2c34fb184a8e684b64b0e4cf930da8ecf1a405c3cf532400477c081863d38b: Status 404 returned error can't find the container with id bf2c34fb184a8e684b64b0e4cf930da8ecf1a405c3cf532400477c081863d38b
Sep 10 12:44:22.903535 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:22.903504    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"]
Sep 10 12:44:22.904888 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:44:22.904859    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7758f874_e331_46b3_8671_551a8ff7d8ad.slice/crio-a75bd70b841cc0424f35e983755c26f47afd1466799e535975b72a781a070360 WatchSource:0}: Error finding container a75bd70b841cc0424f35e983755c26f47afd1466799e535975b72a781a070360: Status 404 returned error can't find the container with id a75bd70b841cc0424f35e983755c26f47afd1466799e535975b72a781a070360
Sep 10 12:44:23.650128 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:23.650046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj" event={"ID":"7758f874-e331-46b3-8671-551a8ff7d8ad","Type":"ContainerStarted","Data":"a75bd70b841cc0424f35e983755c26f47afd1466799e535975b72a781a070360"}
Sep 10 12:44:23.652853 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:23.652816    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4" event={"ID":"b714fed1-f0de-45b1-90a9-344d9268ad10","Type":"ContainerStarted","Data":"bf2c34fb184a8e684b64b0e4cf930da8ecf1a405c3cf532400477c081863d38b"}
Sep 10 12:44:25.672153 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:25.672106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj" event={"ID":"7758f874-e331-46b3-8671-551a8ff7d8ad","Type":"ContainerStarted","Data":"ef581eee457e1918d2f1beb99daa18a43e74d6f351654a9659c15feb03e4aa46"}
Sep 10 12:44:25.672895 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:25.672858    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"
Sep 10 12:44:25.674005 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:25.673978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4" event={"ID":"b714fed1-f0de-45b1-90a9-344d9268ad10","Type":"ContainerStarted","Data":"430d6dfbca2e65595441f06603a90bae76e34f6248b92ac3e7e834f5749f6503"}
Sep 10 12:44:25.674147 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:25.674134    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"
Sep 10 12:44:25.736354 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:25.736296    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj" podStartSLOduration=1.770807122 podStartE2EDuration="3.736277418s" totalImagesPullingTime="1.965470296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:25.735762309 +0000 UTC m=+839.660673794" watchObservedRunningTime="2026-09-10 12:44:25.736277418 +0000 UTC m=+839.661188906"
Sep 10 12:44:25.790344 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:25.790284    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4" podStartSLOduration=1.6978846779999999 podStartE2EDuration="3.790260755s" totalImagesPullingTime="2.092376077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:25.789834707 +0000 UTC m=+839.714746238" watchObservedRunningTime="2026-09-10 12:44:25.790260755 +0000 UTC m=+839.715172244"
Sep 10 12:44:26.492027 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:26.491991    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 12:44:26.492229 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:26.492064    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 12:44:26.681430 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:26.681393    2563 generic.go:356] "Generic (PLEG): container finished" podID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerID="8ee4255861bd0688cdbd3f46ee1af8a44b7ef62574466fb9f10072d21f507857" exitCode=0
Sep 10 12:44:26.681955 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:26.681469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" event={"ID":"f085c5f2-b9da-47e9-9342-7bd37db0b1f9","Type":"ContainerDied","Data":"8ee4255861bd0688cdbd3f46ee1af8a44b7ef62574466fb9f10072d21f507857"}
Sep 10 12:44:26.682113 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:26.682093    2563 scope.go:117] "RemoveContainer" containerID="8ee4255861bd0688cdbd3f46ee1af8a44b7ef62574466fb9f10072d21f507857"
Sep 10 12:44:26.683157 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:26.683128    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:26.683213 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:26.683200    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:26.683250 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:26.683234    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:27.689682 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:27.689643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" event={"ID":"f085c5f2-b9da-47e9-9342-7bd37db0b1f9","Type":"ContainerStarted","Data":"23d4773a142376af39019d1395576751bef5962061f73bcbbcd3611eb5b1a574"}
Sep 10 12:44:28.054903 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:28.054790    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:44:28.055381 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:28.055351    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 12:44:28.055610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:28.055581    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 12:44:29.054639 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:29.054603    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 12:44:29.055258 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:29.055222    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 12:44:30.054343 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:30.054302    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 12:44:30.054646 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:30.054615    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 12:44:31.056169 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:31.056121    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 12:44:31.056690 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:31.056195    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 12:44:32.054152 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:32.054107    2563 patch_prober.go:28] interesting pod/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 12:44:32.054476 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:32.054445    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 12:44:33.163193 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.163159    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:44:33.164305 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.164277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:44:33.172478 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.172450    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:44:33.638507 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.638474    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"]
Sep 10 12:44:33.641300 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.641280    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.649778 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.649755    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-714b74-openshift-default-dockercfg-8ntxt\""
Sep 10 12:44:33.700158 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.700124    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"]
Sep 10 12:44:33.758164 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.758112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-envoy\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.758357 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.758169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istiod-ca-cert\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.758357 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.758222    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-podinfo\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.758357 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.758269    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-workload-socket\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.758357 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.758293    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-credential-socket\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.758357 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.758315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-workload-certs\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.758701 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.758381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-token\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.758701 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.758425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lwd\" (UniqueName: \"kubernetes.io/projected/4d4af67b-c9b1-4075-a28e-c7151d5094a2-kube-api-access-q5lwd\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.758701 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.758484    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-data\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.859989 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.859949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-workload-socket\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.860678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.860652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-credential-socket\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.861757 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.860852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-workload-certs\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.861757 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.860931    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-token\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.861757 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.860974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lwd\" (UniqueName: \"kubernetes.io/projected/4d4af67b-c9b1-4075-a28e-c7151d5094a2-kube-api-access-q5lwd\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.861757 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.861050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-data\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.861757 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.861199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-envoy\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.861757 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.861237    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istiod-ca-cert\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.861757 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.861291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-podinfo\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.864859 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.862451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-data\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.864859 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.860603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-workload-socket\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.864859 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.862745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-credential-socket\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.864859 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.862968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-workload-certs\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.865770 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.865748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istiod-ca-cert\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.869045 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.868991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-envoy\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.871339 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.871286    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-podinfo\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.882209 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.882161    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lwd\" (UniqueName: \"kubernetes.io/projected/4d4af67b-c9b1-4075-a28e-c7151d5094a2-kube-api-access-q5lwd\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.889115 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.889019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-token\") pod \"e2e-inf-b-714b74-openshift-default-687644b978-9bsvv\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") " pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:33.957679 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:33.957635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:34.300876 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:34.300848    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"]
Sep 10 12:44:34.302861 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:44:34.302833    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4af67b_c9b1_4075_a28e_c7151d5094a2.slice/crio-0355c1196c772c7ce77282419cf469421202ffad3f711a9f67fec5869ad71fc4 WatchSource:0}: Error finding container 0355c1196c772c7ce77282419cf469421202ffad3f711a9f67fec5869ad71fc4: Status 404 returned error can't find the container with id 0355c1196c772c7ce77282419cf469421202ffad3f711a9f67fec5869ad71fc4
Sep 10 12:44:34.305243 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:34.305207    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:34.305338 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:34.305282    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:34.305338 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:34.305328    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:34.744750 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:34.744698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv" event={"ID":"4d4af67b-c9b1-4075-a28e-c7151d5094a2","Type":"ContainerStarted","Data":"d13bc78fa9c5e5cf746dec3335d10a929dda305f2b1fc1b41a8636405cbad05d"}
Sep 10 12:44:34.744951 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:34.744779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv" event={"ID":"4d4af67b-c9b1-4075-a28e-c7151d5094a2","Type":"ContainerStarted","Data":"0355c1196c772c7ce77282419cf469421202ffad3f711a9f67fec5869ad71fc4"}
Sep 10 12:44:34.805244 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:34.805181    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv" podStartSLOduration=1.805164043 podStartE2EDuration="1.805164043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:34.803822592 +0000 UTC m=+848.728734083" watchObservedRunningTime="2026-09-10 12:44:34.805164043 +0000 UTC m=+848.730075531"
Sep 10 12:44:34.958780 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:34.958738    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:35.965483 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:35.965437    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:36.582787 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:36.582724    2563 patch_prober.go:28] interesting pod/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 12:44:36.583001 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:36.582825    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 12:44:36.684473 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:36.684426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"
Sep 10 12:44:36.763171 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:36.763094    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:36.764303 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:36.764279    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:37.692135 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:37.692112    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"
Sep 10 12:44:37.771523 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:37.771487    2563 generic.go:356] "Generic (PLEG): container finished" podID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerID="f89fce842d8ec0d4aa82c7d8ae0c54d942f006b8a822febbfde885cd25aeb97f" exitCode=0
Sep 10 12:44:37.771717 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:37.771583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" event={"ID":"91e9c312-8b60-41ce-8bfe-5307b08a77c3","Type":"ContainerDied","Data":"f89fce842d8ec0d4aa82c7d8ae0c54d942f006b8a822febbfde885cd25aeb97f"}
Sep 10 12:44:37.772220 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:37.772198    2563 scope.go:117] "RemoveContainer" containerID="f89fce842d8ec0d4aa82c7d8ae0c54d942f006b8a822febbfde885cd25aeb97f"
Sep 10 12:44:37.773708 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:37.773677    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:37.773798 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:37.773749    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:37.773798 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:37.773785    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:38.779348 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:38.779305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" event={"ID":"91e9c312-8b60-41ce-8bfe-5307b08a77c3","Type":"ContainerStarted","Data":"77264db0e7f4ca15077e08b246dd2857f74a0c2b8e1985635b98573c510856d2"}
Sep 10 12:44:38.780958 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:38.780933    2563 generic.go:356] "Generic (PLEG): container finished" podID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerID="b7fcfd079e16fcca5e3a91a47ce18e861e9ceadbda05d6f1c493c6931ed4ce09" exitCode=0
Sep 10 12:44:38.781061 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:38.781006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" event={"ID":"b3667f65-20e0-4574-9440-89862bfe9d5c","Type":"ContainerDied","Data":"b7fcfd079e16fcca5e3a91a47ce18e861e9ceadbda05d6f1c493c6931ed4ce09"}
Sep 10 12:44:38.781428 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:38.781412    2563 scope.go:117] "RemoveContainer" containerID="b7fcfd079e16fcca5e3a91a47ce18e861e9ceadbda05d6f1c493c6931ed4ce09"
Sep 10 12:44:38.782569 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:38.782527    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:38.782675 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:38.782623    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:38.782741 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:38.782670    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:39.480657 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:39.480143    2563 patch_prober.go:28] interesting pod/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 12:44:39.480657 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:39.480249    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 12:44:39.480657 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:39.480580    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:39.489924 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:39.489888    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:44:39.793455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:39.793357    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" event={"ID":"b3667f65-20e0-4574-9440-89862bfe9d5c","Type":"ContainerStarted","Data":"f9c4597fa2228e20cace0767699d8919caef6639beb574f15c5a1afb76f78c03"}
Sep 10 12:44:40.480299 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:40.480253    2563 patch_prober.go:28] interesting pod/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 12:44:40.480508 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:40.480324    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 12:44:40.764052 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:40.763957    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:40.765014 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:40.764985    2563 patch_prober.go:28] interesting pod/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 10 12:44:40.765282 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:40.765255    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 10 12:44:41.479879 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:41.479838    2563 patch_prober.go:28] interesting pod/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 12:44:41.480477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:41.480442    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 12:44:41.762604 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:41.762477    2563 patch_prober.go:28] interesting pod/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 10 12:44:41.762604 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:41.762565    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 10 12:44:42.481179 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:42.480287    2563 patch_prober.go:28] interesting pod/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 12:44:42.481179 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:42.480359    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 12:44:42.762785 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:42.762209    2563 patch_prober.go:28] interesting pod/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 10 12:44:42.762785 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:42.762289    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 10 12:44:43.479719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:43.479673    2563 patch_prober.go:28] interesting pod/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 12:44:43.479935 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:43.479743    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 12:44:43.587514 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:43.587465    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:43.762380 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:43.762272    2563 patch_prober.go:28] interesting pod/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 10 12:44:43.762380 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:43.762350    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 10 12:44:44.479920 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:44.479880    2563 patch_prober.go:28] interesting pod/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 10 12:44:44.480220 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:44.480189    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 10 12:44:44.762497 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:44.762401    2563 patch_prober.go:28] interesting pod/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 10 12:44:44.763088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:44.763052    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 10 12:44:45.483788 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:45.483757    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:45.484121 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:45.484099    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:45.484737 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:45.484717    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:44:45.766847 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:45.766759    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:45.767826 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:45.767806    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:44:48.849948 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:48.849916    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq"]
Sep 10 12:44:48.853592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:48.853570    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq"
Sep 10 12:44:48.856888 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:48.856863    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:44:48.858097 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:48.858082    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:44:48.868055 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:48.868032    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq"]
Sep 10 12:44:48.959058 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:48.959018    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c8b03106-445lq\" (UID: \"a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq"
Sep 10 12:44:49.060047 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:49.059998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c8b03106-445lq\" (UID: \"a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq"
Sep 10 12:44:49.061340 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:49.061122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-c8b03106-445lq\" (UID: \"a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq"
Sep 10 12:44:49.165429 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:49.165337    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq"
Sep 10 12:44:49.448939 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:49.448897    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq"]
Sep 10 12:44:49.457694 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:44:49.457664    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3db8937_fc0c_407d_bfe6_c04a8bcc3fa2.slice/crio-ad5132fc898310ed29f2c49d9580771fba46cfc83fe3ba65cd6b73d0c0123b36 WatchSource:0}: Error finding container ad5132fc898310ed29f2c49d9580771fba46cfc83fe3ba65cd6b73d0c0123b36: Status 404 returned error can't find the container with id ad5132fc898310ed29f2c49d9580771fba46cfc83fe3ba65cd6b73d0c0123b36
Sep 10 12:44:49.861134 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:49.861028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq" event={"ID":"a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2","Type":"ContainerStarted","Data":"ad5132fc898310ed29f2c49d9580771fba46cfc83fe3ba65cd6b73d0c0123b36"}
Sep 10 12:44:52.892442 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:52.892158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq" event={"ID":"a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2","Type":"ContainerStarted","Data":"5d53b9d27e3fa1c8b28b37904e3a94691091ca770525217228e28336ae05ec81"}
Sep 10 12:44:52.914421 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:52.914198    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq" podStartSLOduration=2.070715807 podStartE2EDuration="4.914179362s" totalImagesPullingTime="2.843463555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:52.913585465 +0000 UTC m=+866.838496944" watchObservedRunningTime="2026-09-10 12:44:52.914179362 +0000 UTC m=+866.839090851"
Sep 10 12:44:53.900902 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:53.900867    2563 generic.go:356] "Generic (PLEG): container finished" podID="a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2" containerID="5d53b9d27e3fa1c8b28b37904e3a94691091ca770525217228e28336ae05ec81" exitCode=0
Sep 10 12:44:53.901262 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:53.900944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq" event={"ID":"a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2","Type":"ContainerDied","Data":"5d53b9d27e3fa1c8b28b37904e3a94691091ca770525217228e28336ae05ec81"}
Sep 10 12:44:55.057683 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:55.057656    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq"
Sep 10 12:44:55.127909 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:55.127876    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2-maas-api-service-ca\") pod \"a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2\" (UID: \"a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2\") "
Sep 10 12:44:55.128322 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:55.128295    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2" (UID: "a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:44:55.229290 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:55.229247    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2-maas-api-service-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:44:55.915187 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:55.915149    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq"
Sep 10 12:44:55.915352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:55.915194    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq" event={"ID":"a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2","Type":"ContainerDied","Data":"ad5132fc898310ed29f2c49d9580771fba46cfc83fe3ba65cd6b73d0c0123b36"}
Sep 10 12:44:55.915352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:55.915252    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad5132fc898310ed29f2c49d9580771fba46cfc83fe3ba65cd6b73d0c0123b36"
Sep 10 12:44:57.929415 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:57.929325    2563 generic.go:356] "Generic (PLEG): container finished" podID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerID="d13bc78fa9c5e5cf746dec3335d10a929dda305f2b1fc1b41a8636405cbad05d" exitCode=0
Sep 10 12:44:57.929834 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:57.929401    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv" event={"ID":"4d4af67b-c9b1-4075-a28e-c7151d5094a2","Type":"ContainerDied","Data":"d13bc78fa9c5e5cf746dec3335d10a929dda305f2b1fc1b41a8636405cbad05d"}
Sep 10 12:44:57.929905 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:57.929890    2563 scope.go:117] "RemoveContainer" containerID="d13bc78fa9c5e5cf746dec3335d10a929dda305f2b1fc1b41a8636405cbad05d"
Sep 10 12:44:57.931163 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:57.931132    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:57.931235 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:57.931205    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:57.931274 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:57.931243    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:44:58.523751 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:58.523703    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:44:58.936791 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:58.936756    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv" event={"ID":"4d4af67b-c9b1-4075-a28e-c7151d5094a2","Type":"ContainerStarted","Data":"07af6887d518edb463e25e6f6b12e2bd5deb6c6a3086a8f23f0556dbb21daa3b"}
Sep 10 12:44:58.959100 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:58.959038    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:44:59.625108 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:59.625076    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"]
Sep 10 12:44:59.625315 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:59.625279    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj" podUID="7758f874-e331-46b3-8671-551a8ff7d8ad" containerName="payload-processing" containerID="cri-o://ef581eee457e1918d2f1beb99daa18a43e74d6f351654a9659c15feb03e4aa46" gracePeriod=30
Sep 10 12:44:59.631726 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:59.631700    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"]
Sep 10 12:44:59.632336 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:59.632106    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4" podUID="b714fed1-f0de-45b1-90a9-344d9268ad10" containerName="payload-pre-processing" containerID="cri-o://430d6dfbca2e65595441f06603a90bae76e34f6248b92ac3e7e834f5749f6503" gracePeriod=30
Sep 10 12:44:59.870655 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:59.870612    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq"]
Sep 10 12:44:59.879889 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:59.879810    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c8b03106-445lq"]
Sep 10 12:44:59.974584 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:59.962613    2563 patch_prober.go:28] interesting pod/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 10 12:44:59.974584 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:59.962682    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv" podUID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 10 12:45:00.006246 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:59.990969    2563 generic.go:356] "Generic (PLEG): container finished" podID="7758f874-e331-46b3-8671-551a8ff7d8ad" containerID="ef581eee457e1918d2f1beb99daa18a43e74d6f351654a9659c15feb03e4aa46" exitCode=0
Sep 10 12:45:00.006246 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:59.991130    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj" event={"ID":"7758f874-e331-46b3-8671-551a8ff7d8ad","Type":"ContainerDied","Data":"ef581eee457e1918d2f1beb99daa18a43e74d6f351654a9659c15feb03e4aa46"}
Sep 10 12:45:00.006246 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:59.993937    2563 generic.go:356] "Generic (PLEG): container finished" podID="b714fed1-f0de-45b1-90a9-344d9268ad10" containerID="430d6dfbca2e65595441f06603a90bae76e34f6248b92ac3e7e834f5749f6503" exitCode=0
Sep 10 12:45:00.006246 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:44:59.995597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4" event={"ID":"b714fed1-f0de-45b1-90a9-344d9268ad10","Type":"ContainerDied","Data":"430d6dfbca2e65595441f06603a90bae76e34f6248b92ac3e7e834f5749f6503"}
Sep 10 12:45:00.013917 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.013887    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"]
Sep 10 12:45:00.014315 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.014286    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerName="istio-proxy" containerID="cri-o://f9c4597fa2228e20cace0767699d8919caef6639beb574f15c5a1afb76f78c03" gracePeriod=30
Sep 10 12:45:00.022773 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.022747    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"
Sep 10 12:45:00.026155 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.026137    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"
Sep 10 12:45:00.036939 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.036902    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"]
Sep 10 12:45:00.037394 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.037382    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2" containerName="revoke-keys"
Sep 10 12:45:00.037442 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.037396    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2" containerName="revoke-keys"
Sep 10 12:45:00.037442 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.037418    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7758f874-e331-46b3-8671-551a8ff7d8ad" containerName="payload-processing"
Sep 10 12:45:00.037442 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.037424    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7758f874-e331-46b3-8671-551a8ff7d8ad" containerName="payload-processing"
Sep 10 12:45:00.037442 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.037439    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b714fed1-f0de-45b1-90a9-344d9268ad10" containerName="payload-pre-processing"
Sep 10 12:45:00.037586 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.037445    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b714fed1-f0de-45b1-90a9-344d9268ad10" containerName="payload-pre-processing"
Sep 10 12:45:00.037586 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.037516    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2" containerName="revoke-keys"
Sep 10 12:45:00.037586 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.037526    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7758f874-e331-46b3-8671-551a8ff7d8ad" containerName="payload-processing"
Sep 10 12:45:00.037586 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.037534    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b714fed1-f0de-45b1-90a9-344d9268ad10" containerName="payload-pre-processing"
Sep 10 12:45:00.039790 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.039769    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.042979 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.042318    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1312ea\"/\"openshift-service-ca.crt\""
Sep 10 12:45:00.042979 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.042503    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-1312ea\"/\"kube-root-ca.crt\""
Sep 10 12:45:00.042979 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.042727    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1312ea\"/\"default-dockercfg-q5q96\""
Sep 10 12:45:00.042979 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.042791    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1312ea\"/\"simulator-w4-1312ea-kserve-self-signed-certs\""
Sep 10 12:45:00.085592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.079852    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"]
Sep 10 12:45:00.085592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.084820    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b714fed1-f0de-45b1-90a9-344d9268ad10-plugins-config-volume\") pod \"b714fed1-f0de-45b1-90a9-344d9268ad10\" (UID: \"b714fed1-f0de-45b1-90a9-344d9268ad10\") "
Sep 10 12:45:00.085592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.084907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbp6f\" (UniqueName: \"kubernetes.io/projected/7758f874-e331-46b3-8671-551a8ff7d8ad-kube-api-access-gbp6f\") pod \"7758f874-e331-46b3-8671-551a8ff7d8ad\" (UID: \"7758f874-e331-46b3-8671-551a8ff7d8ad\") "
Sep 10 12:45:00.085592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.084955    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7758f874-e331-46b3-8671-551a8ff7d8ad-plugins-config-volume\") pod \"7758f874-e331-46b3-8671-551a8ff7d8ad\" (UID: \"7758f874-e331-46b3-8671-551a8ff7d8ad\") "
Sep 10 12:45:00.085592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.085028    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffstb\" (UniqueName: \"kubernetes.io/projected/b714fed1-f0de-45b1-90a9-344d9268ad10-kube-api-access-ffstb\") pod \"b714fed1-f0de-45b1-90a9-344d9268ad10\" (UID: \"b714fed1-f0de-45b1-90a9-344d9268ad10\") "
Sep 10 12:45:00.085592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.085277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-home\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.085592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.085327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144c7178-a13b-4068-8866-5c0592344252-tls-certs\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.085592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.085428    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-dshm\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.085592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.085455    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4prf\" (UniqueName: \"kubernetes.io/projected/144c7178-a13b-4068-8866-5c0592344252-kube-api-access-l4prf\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.085592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.085580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-tmp-dir\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.086284 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.085613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-model-cache\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.087205 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.087147    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b714fed1-f0de-45b1-90a9-344d9268ad10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b714fed1-f0de-45b1-90a9-344d9268ad10" (UID: "b714fed1-f0de-45b1-90a9-344d9268ad10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:45:00.087668 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.087587    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7758f874-e331-46b3-8671-551a8ff7d8ad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7758f874-e331-46b3-8671-551a8ff7d8ad" (UID: "7758f874-e331-46b3-8671-551a8ff7d8ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:45:00.091664 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.091612    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b714fed1-f0de-45b1-90a9-344d9268ad10-kube-api-access-ffstb" (OuterVolumeSpecName: "kube-api-access-ffstb") pod "b714fed1-f0de-45b1-90a9-344d9268ad10" (UID: "b714fed1-f0de-45b1-90a9-344d9268ad10"). InnerVolumeSpecName "kube-api-access-ffstb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:45:00.091851 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.091644    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7758f874-e331-46b3-8671-551a8ff7d8ad-kube-api-access-gbp6f" (OuterVolumeSpecName: "kube-api-access-gbp6f") pod "7758f874-e331-46b3-8671-551a8ff7d8ad" (UID: "7758f874-e331-46b3-8671-551a8ff7d8ad"). InnerVolumeSpecName "kube-api-access-gbp6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:45:00.174827 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.174786    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817405-bcqjp"]
Sep 10 12:45:00.186311 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.186273    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-home\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.186622 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.186539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144c7178-a13b-4068-8866-5c0592344252-tls-certs\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.186820 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.186804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-dshm\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.186924 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.186909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4prf\" (UniqueName: \"kubernetes.io/projected/144c7178-a13b-4068-8866-5c0592344252-kube-api-access-l4prf\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.187090 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.187076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-tmp-dir\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.187194 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.187182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-model-cache\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.187340 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.187327    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7758f874-e331-46b3-8671-551a8ff7d8ad-plugins-config-volume\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:00.187437 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.187427    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffstb\" (UniqueName: \"kubernetes.io/projected/b714fed1-f0de-45b1-90a9-344d9268ad10-kube-api-access-ffstb\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:00.187520 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.187509    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b714fed1-f0de-45b1-90a9-344d9268ad10-plugins-config-volume\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:00.187621 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.187609    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbp6f\" (UniqueName: \"kubernetes.io/projected/7758f874-e331-46b3-8671-551a8ff7d8ad-kube-api-access-gbp6f\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:00.188094 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.188070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-model-cache\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.188452 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.188433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-home\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.189934 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.189916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-bcqjp"
Sep 10 12:45:00.191574 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.190424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-tmp-dir\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.198770 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.198742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-dshm\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.199901 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.199790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144c7178-a13b-4068-8866-5c0592344252-tls-certs\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.203175 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.200813    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5"]
Sep 10 12:45:00.204647 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.202808    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:45:00.207627 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.206883    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5abff0-29817405-79lnd"]
Sep 10 12:45:00.211705 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.210442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5"
Sep 10 12:45:00.213126 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.213106    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817405-bcqjp"]
Sep 10 12:45:00.213399 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.213381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5abff0-29817405-79lnd"
Sep 10 12:45:00.214755 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.214459    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-bcqjp"
Sep 10 12:45:00.214755 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.214616    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5"]
Sep 10 12:45:00.217151 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.217126    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5abff0-29817405-79lnd"]
Sep 10 12:45:00.220650 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.220601    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4prf\" (UniqueName: \"kubernetes.io/projected/144c7178-a13b-4068-8866-5c0592344252-kube-api-access-l4prf\") pod \"simulator-w4-1312ea-kserve-5f558bb894-5ntdk\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") " pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.227811 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.227792    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5"
Sep 10 12:45:00.235697 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.235675    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5abff0-29817405-79lnd"
Sep 10 12:45:00.378818 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.378388    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:00.575153 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.574651    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817405-bcqjp"]
Sep 10 12:45:00.579363 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.579301    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5"]
Sep 10 12:45:00.583197 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:45:00.582970    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93b621bd_daf2_4dea_b31c_62b6c73fea8a.slice/crio-f7a9edf362f4b9adc5d59b00716bfd50234af074efb302109451c6b4d06c4e22 WatchSource:0}: Error finding container f7a9edf362f4b9adc5d59b00716bfd50234af074efb302109451c6b4d06c4e22: Status 404 returned error can't find the container with id f7a9edf362f4b9adc5d59b00716bfd50234af074efb302109451c6b4d06c4e22
Sep 10 12:45:00.583935 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.583682    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5abff0-29817405-79lnd"]
Sep 10 12:45:00.682362 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.681946    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"]
Sep 10 12:45:00.686741 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:45:00.686676    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod144c7178_a13b_4068_8866_5c0592344252.slice/crio-7c6bfd1ce9d452a2f771aa4878e4cf9d9de25517eca9cc605afe17b331f2a83a WatchSource:0}: Error finding container 7c6bfd1ce9d452a2f771aa4878e4cf9d9de25517eca9cc605afe17b331f2a83a: Status 404 returned error can't find the container with id 7c6bfd1ce9d452a2f771aa4878e4cf9d9de25517eca9cc605afe17b331f2a83a
Sep 10 12:45:00.713544 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.713438    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2" path="/var/lib/kubelet/pods/a3db8937-fc0c-407d-bfe6-c04a8bcc3fa2/volumes"
Sep 10 12:45:00.776717 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.776684    2563 patch_prober.go:28] interesting pod/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 12:45:00.777003 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.776963    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 12:45:00.960621 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.960149    2563 patch_prober.go:28] interesting pod/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 10 12:45:00.960621 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.960214    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv" podUID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 10 12:45:00.991922 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.991891    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"]
Sep 10 12:45:00.997527 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.996893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:00.999635 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:00.999611    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-760a0d05-gw-openshift-default-dockercfg-9pqb8\""
Sep 10 12:45:01.007268 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.007238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-bcqjp" event={"ID":"2da7d2c3-3e1f-4736-ac40-7f2a49ed8f8c","Type":"ContainerStarted","Data":"051288599b3a1662bb959cd89e27379f6e4ff4f0b0756df77d6ba72a7224f492"}
Sep 10 12:45:01.007507 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.007485    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-bcqjp" event={"ID":"2da7d2c3-3e1f-4736-ac40-7f2a49ed8f8c","Type":"ContainerStarted","Data":"d2ca1696ce430bd92ec6f280d87a9fe0eed9d145bebf1947a6c9aa3949ad887d"}
Sep 10 12:45:01.010796 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.010775    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"
Sep 10 12:45:01.011483 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.010778    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj" event={"ID":"7758f874-e331-46b3-8671-551a8ff7d8ad","Type":"ContainerDied","Data":"a75bd70b841cc0424f35e983755c26f47afd1466799e535975b72a781a070360"}
Sep 10 12:45:01.011483 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.011365    2563 scope.go:117] "RemoveContainer" containerID="ef581eee457e1918d2f1beb99daa18a43e74d6f351654a9659c15feb03e4aa46"
Sep 10 12:45:01.013322 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.013300    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"]
Sep 10 12:45:01.015516 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.015449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk" event={"ID":"144c7178-a13b-4068-8866-5c0592344252","Type":"ContainerStarted","Data":"f31160967042beea125449a8a5c4513456fb7986765d92fa8b830e807c2b8601"}
Sep 10 12:45:01.015828 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.015758    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk" event={"ID":"144c7178-a13b-4068-8866-5c0592344252","Type":"ContainerStarted","Data":"7c6bfd1ce9d452a2f771aa4878e4cf9d9de25517eca9cc605afe17b331f2a83a"}
Sep 10 12:45:01.017716 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.017688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5abff0-29817405-79lnd" event={"ID":"93b621bd-daf2-4dea-b31c-62b6c73fea8a","Type":"ContainerStarted","Data":"f7a9edf362f4b9adc5d59b00716bfd50234af074efb302109451c6b4d06c4e22"}
Sep 10 12:45:01.019951 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.019936    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"
Sep 10 12:45:01.020073 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.019995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4" event={"ID":"b714fed1-f0de-45b1-90a9-344d9268ad10","Type":"ContainerDied","Data":"bf2c34fb184a8e684b64b0e4cf930da8ecf1a405c3cf532400477c081863d38b"}
Sep 10 12:45:01.021984 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.021963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5" event={"ID":"0c202854-fa69-421e-9bdc-a1458d773562","Type":"ContainerStarted","Data":"4dd16cc166fcd819906ac3ae9e4bafd1e8a26217bcffead4c813c9129703768e"}
Sep 10 12:45:01.022147 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.022129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5" event={"ID":"0c202854-fa69-421e-9bdc-a1458d773562","Type":"ContainerStarted","Data":"9553651a53ef05fffa8f8c19ee2c8f74bbdd94c7468e71675d3a61d886b331d1"}
Sep 10 12:45:01.027134 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.027115    2563 scope.go:117] "RemoveContainer" containerID="430d6dfbca2e65595441f06603a90bae76e34f6248b92ac3e7e834f5749f6503"
Sep 10 12:45:01.091002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.090934    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5abff0-29817405-79lnd" podStartSLOduration=1.090917207 podStartE2EDuration="1.090917207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:01.089245413 +0000 UTC m=+875.014156894" watchObservedRunningTime="2026-09-10 12:45:01.090917207 +0000 UTC m=+875.015828696"
Sep 10 12:45:01.097888 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.097857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-workload-certs\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.097888 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.097892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istiod-ca-cert\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.098060 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.097982    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv46j\" (UniqueName: \"kubernetes.io/projected/5d8e5d8b-66af-46b4-92bd-094288d80ba9-kube-api-access-sv46j\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.098060 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.098008    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-podinfo\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.098130 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.098068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-data\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.098130 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.098116    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-workload-socket\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.098191 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.098156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-envoy\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.098226 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.098192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-credential-socket\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.098226 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.098210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-token\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.113273 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.113220    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk" podStartSLOduration=1.113203153 podStartE2EDuration="1.113203153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:01.110184492 +0000 UTC m=+875.035095981" watchObservedRunningTime="2026-09-10 12:45:01.113203153 +0000 UTC m=+875.038114641"
Sep 10 12:45:01.138331 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.138297    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"]
Sep 10 12:45:01.144779 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.144749    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c8b03106-77d5898bb5-wr6gj"]
Sep 10 12:45:01.176587 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.175916    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"]
Sep 10 12:45:01.178030 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.177811    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c8b03106-74f998d46b-pcls4"]
Sep 10 12:45:01.186487 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.186414    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-bcqjp" podStartSLOduration=1.18639637 podStartE2EDuration="1.18639637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:01.183025067 +0000 UTC m=+875.107936555" watchObservedRunningTime="2026-09-10 12:45:01.18639637 +0000 UTC m=+875.111307854"
Sep 10 12:45:01.199517 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.199485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv46j\" (UniqueName: \"kubernetes.io/projected/5d8e5d8b-66af-46b4-92bd-094288d80ba9-kube-api-access-sv46j\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.199715 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.199606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-podinfo\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.199715 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.199688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-data\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.199849 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.199742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-workload-socket\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.199849 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.199801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-envoy\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.199953 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.199845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-credential-socket\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.199953 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.199874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-token\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.199953 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.199913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-workload-certs\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.199953 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.199939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istiod-ca-cert\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.200379 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.200357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-data\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.200591 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.200547    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-credential-socket\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.200727 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.200697    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-workload-socket\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.200899 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.200756    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istiod-ca-cert\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.200962 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.200812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-workload-certs\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.202072 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.202023    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5" podStartSLOduration=1.202007566 podStartE2EDuration="1.202007566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:01.199102484 +0000 UTC m=+875.124013982" watchObservedRunningTime="2026-09-10 12:45:01.202007566 +0000 UTC m=+875.126919056"
Sep 10 12:45:01.202405 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.202378    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-podinfo\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.202500 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.202484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-envoy\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.211074 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.210968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-token\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.214235 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.214211    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv46j\" (UniqueName: \"kubernetes.io/projected/5d8e5d8b-66af-46b4-92bd-094288d80ba9-kube-api-access-sv46j\") pod \"e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") " pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.310908 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.310864    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:01.618342 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.618316    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"]
Sep 10 12:45:01.619746 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:45:01.619720    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8e5d8b_66af_46b4_92bd_094288d80ba9.slice/crio-d4ee10000733f69932a8699644310ddbcaacaa4a1bc133bba9d501a92556ec3d WatchSource:0}: Error finding container d4ee10000733f69932a8699644310ddbcaacaa4a1bc133bba9d501a92556ec3d: Status 404 returned error can't find the container with id d4ee10000733f69932a8699644310ddbcaacaa4a1bc133bba9d501a92556ec3d
Sep 10 12:45:01.622201 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.622168    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:45:01.622253 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.622236    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:45:01.622298 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.622264    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:45:01.961141 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.959071    2563 patch_prober.go:28] interesting pod/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 10 12:45:01.961141 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:01.959133    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv" podUID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 10 12:45:02.030989 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.030952    2563 generic.go:356] "Generic (PLEG): container finished" podID="93b621bd-daf2-4dea-b31c-62b6c73fea8a" containerID="487eaaf723b1ffbe249555db09d2c3bd3e1cf520773e7dcf14183b5ee8e3ca10" exitCode=0
Sep 10 12:45:02.031439 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.031028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5abff0-29817405-79lnd" event={"ID":"93b621bd-daf2-4dea-b31c-62b6c73fea8a","Type":"ContainerDied","Data":"487eaaf723b1ffbe249555db09d2c3bd3e1cf520773e7dcf14183b5ee8e3ca10"}
Sep 10 12:45:02.033617 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.033585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" event={"ID":"5d8e5d8b-66af-46b4-92bd-094288d80ba9","Type":"ContainerStarted","Data":"04eae6173d569a4cedea97b58cbe7ef62b12fc74265281368ef5604a5d5eea2b"}
Sep 10 12:45:02.033742 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.033645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" event={"ID":"5d8e5d8b-66af-46b4-92bd-094288d80ba9","Type":"ContainerStarted","Data":"d4ee10000733f69932a8699644310ddbcaacaa4a1bc133bba9d501a92556ec3d"}
Sep 10 12:45:02.035056 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.035030    2563 generic.go:356] "Generic (PLEG): container finished" podID="0c202854-fa69-421e-9bdc-a1458d773562" containerID="4dd16cc166fcd819906ac3ae9e4bafd1e8a26217bcffead4c813c9129703768e" exitCode=0
Sep 10 12:45:02.035181 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.035116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5" event={"ID":"0c202854-fa69-421e-9bdc-a1458d773562","Type":"ContainerDied","Data":"4dd16cc166fcd819906ac3ae9e4bafd1e8a26217bcffead4c813c9129703768e"}
Sep 10 12:45:02.036583 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.036562    2563 generic.go:356] "Generic (PLEG): container finished" podID="2da7d2c3-3e1f-4736-ac40-7f2a49ed8f8c" containerID="051288599b3a1662bb959cd89e27379f6e4ff4f0b0756df77d6ba72a7224f492" exitCode=0
Sep 10 12:45:02.036698 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.036644    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-bcqjp" event={"ID":"2da7d2c3-3e1f-4736-ac40-7f2a49ed8f8c","Type":"ContainerDied","Data":"051288599b3a1662bb959cd89e27379f6e4ff4f0b0756df77d6ba72a7224f492"}
Sep 10 12:45:02.092723 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.091724    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" podStartSLOduration=2.091707174 podStartE2EDuration="2.091707174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:02.089861824 +0000 UTC m=+876.014773312" watchObservedRunningTime="2026-09-10 12:45:02.091707174 +0000 UTC m=+876.016618662"
Sep 10 12:45:02.311241 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.311199    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:02.316262 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.316236    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:02.706249 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.706216    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7758f874-e331-46b3-8671-551a8ff7d8ad" path="/var/lib/kubelet/pods/7758f874-e331-46b3-8671-551a8ff7d8ad/volumes"
Sep 10 12:45:02.706610 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.706595    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b714fed1-f0de-45b1-90a9-344d9268ad10" path="/var/lib/kubelet/pods/b714fed1-f0de-45b1-90a9-344d9268ad10/volumes"
Sep 10 12:45:02.958774 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.958673    2563 patch_prober.go:28] interesting pod/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 10 12:45:02.958774 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:02.958741    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv" podUID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 10 12:45:03.044190 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:03.043796    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:03.044817 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:03.044790    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:03.324039 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:03.324002    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5"
Sep 10 12:45:03.326332 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:03.326311    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-bcqjp"
Sep 10 12:45:03.343337 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:03.343315    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5abff0-29817405-79lnd"
Sep 10 12:45:03.960016 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:03.959977    2563 patch_prober.go:28] interesting pod/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 10 12:45:03.960271 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:03.960237    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv" podUID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 10 12:45:04.062583 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.061278    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5abff0-29817405-79lnd" event={"ID":"93b621bd-daf2-4dea-b31c-62b6c73fea8a","Type":"ContainerDied","Data":"f7a9edf362f4b9adc5d59b00716bfd50234af074efb302109451c6b4d06c4e22"}
Sep 10 12:45:04.062583 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.061349    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7a9edf362f4b9adc5d59b00716bfd50234af074efb302109451c6b4d06c4e22"
Sep 10 12:45:04.062583 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.061431    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5abff0-29817405-79lnd"
Sep 10 12:45:04.091144 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.091108    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5"
Sep 10 12:45:04.091144 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.091119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5" event={"ID":"0c202854-fa69-421e-9bdc-a1458d773562","Type":"ContainerDied","Data":"9553651a53ef05fffa8f8c19ee2c8f74bbdd94c7468e71675d3a61d886b331d1"}
Sep 10 12:45:04.091363 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.091222    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9553651a53ef05fffa8f8c19ee2c8f74bbdd94c7468e71675d3a61d886b331d1"
Sep 10 12:45:04.094310 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.094286    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-bcqjp"
Sep 10 12:45:04.094434 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.094317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-bcqjp" event={"ID":"2da7d2c3-3e1f-4736-ac40-7f2a49ed8f8c","Type":"ContainerDied","Data":"d2ca1696ce430bd92ec6f280d87a9fe0eed9d145bebf1947a6c9aa3949ad887d"}
Sep 10 12:45:04.094434 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.094372    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2ca1696ce430bd92ec6f280d87a9fe0eed9d145bebf1947a6c9aa3949ad887d"
Sep 10 12:45:04.889851 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.889809    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"]
Sep 10 12:45:04.890300 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.890286    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c202854-fa69-421e-9bdc-a1458d773562" containerName="cleanup"
Sep 10 12:45:04.890351 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.890301    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c202854-fa69-421e-9bdc-a1458d773562" containerName="cleanup"
Sep 10 12:45:04.890351 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.890321    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93b621bd-daf2-4dea-b31c-62b6c73fea8a" containerName="cleanup"
Sep 10 12:45:04.890351 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.890327    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="93b621bd-daf2-4dea-b31c-62b6c73fea8a" containerName="cleanup"
Sep 10 12:45:04.890351 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.890335    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2da7d2c3-3e1f-4736-ac40-7f2a49ed8f8c" containerName="cleanup"
Sep 10 12:45:04.890351 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.890341    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2da7d2c3-3e1f-4736-ac40-7f2a49ed8f8c" containerName="cleanup"
Sep 10 12:45:04.890494 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.890400    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c202854-fa69-421e-9bdc-a1458d773562" containerName="cleanup"
Sep 10 12:45:04.890494 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.890412    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="93b621bd-daf2-4dea-b31c-62b6c73fea8a" containerName="cleanup"
Sep 10 12:45:04.890494 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.890419    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2da7d2c3-3e1f-4736-ac40-7f2a49ed8f8c" containerName="cleanup"
Sep 10 12:45:04.893009 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.892991    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"
Sep 10 12:45:04.896213 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.896186    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-760a0d05-dockercfg-xb5xp\""
Sep 10 12:45:04.896348 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.896224    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-760a0d05\""
Sep 10 12:45:04.907788 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.907758    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"]
Sep 10 12:45:04.947828 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.947598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1226061-dc18-41e0-84fd-96e4917eeb9c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26\" (UID: \"c1226061-dc18-41e0-84fd-96e4917eeb9c\") " pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"
Sep 10 12:45:04.947828 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.947714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx8sr\" (UniqueName: \"kubernetes.io/projected/c1226061-dc18-41e0-84fd-96e4917eeb9c-kube-api-access-qx8sr\") pod \"payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26\" (UID: \"c1226061-dc18-41e0-84fd-96e4917eeb9c\") " pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"
Sep 10 12:45:04.967139 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.967095    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:45:04.967694 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.967672    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:45:04.968563 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:04.968530    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:45:05.051290 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.051233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1226061-dc18-41e0-84fd-96e4917eeb9c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26\" (UID: \"c1226061-dc18-41e0-84fd-96e4917eeb9c\") " pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"
Sep 10 12:45:05.051479 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.051331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx8sr\" (UniqueName: \"kubernetes.io/projected/c1226061-dc18-41e0-84fd-96e4917eeb9c-kube-api-access-qx8sr\") pod \"payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26\" (UID: \"c1226061-dc18-41e0-84fd-96e4917eeb9c\") " pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"
Sep 10 12:45:05.052385 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.052353    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1226061-dc18-41e0-84fd-96e4917eeb9c-plugins-config-volume\") pod \"payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26\" (UID: \"c1226061-dc18-41e0-84fd-96e4917eeb9c\") " pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"
Sep 10 12:45:05.061894 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.061864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx8sr\" (UniqueName: \"kubernetes.io/projected/c1226061-dc18-41e0-84fd-96e4917eeb9c-kube-api-access-qx8sr\") pod \"payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26\" (UID: \"c1226061-dc18-41e0-84fd-96e4917eeb9c\") " pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"
Sep 10 12:45:05.203954 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.203916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"
Sep 10 12:45:05.299919 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.299892    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:45:05.354883 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.354833    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-workload-socket\") pod \"b3667f65-20e0-4574-9440-89862bfe9d5c\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") "
Sep 10 12:45:05.355066 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.354890    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-envoy\") pod \"b3667f65-20e0-4574-9440-89862bfe9d5c\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") "
Sep 10 12:45:05.355066 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.354973    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-token\") pod \"b3667f65-20e0-4574-9440-89862bfe9d5c\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") "
Sep 10 12:45:05.355066 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.355042    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3667f65-20e0-4574-9440-89862bfe9d5c-istiod-ca-cert\") pod \"b3667f65-20e0-4574-9440-89862bfe9d5c\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") "
Sep 10 12:45:05.355271 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.355067    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-workload-certs\") pod \"b3667f65-20e0-4574-9440-89862bfe9d5c\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") "
Sep 10 12:45:05.355271 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.355094    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jgc6\" (UniqueName: \"kubernetes.io/projected/b3667f65-20e0-4574-9440-89862bfe9d5c-kube-api-access-6jgc6\") pod \"b3667f65-20e0-4574-9440-89862bfe9d5c\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") "
Sep 10 12:45:05.355271 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.355188    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b3667f65-20e0-4574-9440-89862bfe9d5c" (UID: "b3667f65-20e0-4574-9440-89862bfe9d5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:45:05.355429 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.355380    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b3667f65-20e0-4574-9440-89862bfe9d5c" (UID: "b3667f65-20e0-4574-9440-89862bfe9d5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:45:05.355900 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.355784    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-data\") pod \"b3667f65-20e0-4574-9440-89862bfe9d5c\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") "
Sep 10 12:45:05.355900 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.355881    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3667f65-20e0-4574-9440-89862bfe9d5c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b3667f65-20e0-4574-9440-89862bfe9d5c" (UID: "b3667f65-20e0-4574-9440-89862bfe9d5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:45:05.356110 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.356053    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-data" (OuterVolumeSpecName: "istio-data") pod "b3667f65-20e0-4574-9440-89862bfe9d5c" (UID: "b3667f65-20e0-4574-9440-89862bfe9d5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:45:05.356173 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.356121    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-credential-socket\") pod \"b3667f65-20e0-4574-9440-89862bfe9d5c\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") "
Sep 10 12:45:05.356229 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.356180    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-podinfo\") pod \"b3667f65-20e0-4574-9440-89862bfe9d5c\" (UID: \"b3667f65-20e0-4574-9440-89862bfe9d5c\") "
Sep 10 12:45:05.356805 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.356376    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b3667f65-20e0-4574-9440-89862bfe9d5c" (UID: "b3667f65-20e0-4574-9440-89862bfe9d5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:45:05.357228 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.357206    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-workload-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:05.357339 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.357326    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3667f65-20e0-4574-9440-89862bfe9d5c-istiod-ca-cert\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:05.357461 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.357448    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-workload-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:05.357581 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.357568    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-data\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:05.357678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.357664    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-credential-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:05.358257 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.358224    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-token" (OuterVolumeSpecName: "istio-token") pod "b3667f65-20e0-4574-9440-89862bfe9d5c" (UID: "b3667f65-20e0-4574-9440-89862bfe9d5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:45:05.358611 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.358580    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3667f65-20e0-4574-9440-89862bfe9d5c-kube-api-access-6jgc6" (OuterVolumeSpecName: "kube-api-access-6jgc6") pod "b3667f65-20e0-4574-9440-89862bfe9d5c" (UID: "b3667f65-20e0-4574-9440-89862bfe9d5c"). InnerVolumeSpecName "kube-api-access-6jgc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:45:05.358716 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.358632    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b3667f65-20e0-4574-9440-89862bfe9d5c" (UID: "b3667f65-20e0-4574-9440-89862bfe9d5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:45:05.358947 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.358915    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b3667f65-20e0-4574-9440-89862bfe9d5c" (UID: "b3667f65-20e0-4574-9440-89862bfe9d5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:45:05.458581 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.458502    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-podinfo\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:05.458581 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.458542    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-envoy\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:05.458581 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.458586    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3667f65-20e0-4574-9440-89862bfe9d5c-istio-token\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:05.458877 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.458595    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jgc6\" (UniqueName: \"kubernetes.io/projected/b3667f65-20e0-4574-9440-89862bfe9d5c-kube-api-access-6jgc6\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:05.509153 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:05.508954    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"]
Sep 10 12:45:05.512087 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:45:05.512054    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1226061_dc18_41e0_84fd_96e4917eeb9c.slice/crio-7b372d837ec337f1fbc3a108080952485e9fbeb47e26b48e5b2f4942b1154ee5 WatchSource:0}: Error finding container 7b372d837ec337f1fbc3a108080952485e9fbeb47e26b48e5b2f4942b1154ee5: Status 404 returned error can't find the container with id 7b372d837ec337f1fbc3a108080952485e9fbeb47e26b48e5b2f4942b1154ee5
Sep 10 12:45:06.108693 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.108651    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26" event={"ID":"c1226061-dc18-41e0-84fd-96e4917eeb9c","Type":"ContainerStarted","Data":"51d7ebabce49ce3c2d676c7572866d935c3aecdf72ba5652445ea1ffc15f4ddd"}
Sep 10 12:45:06.108895 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.108714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26" event={"ID":"c1226061-dc18-41e0-84fd-96e4917eeb9c","Type":"ContainerStarted","Data":"7b372d837ec337f1fbc3a108080952485e9fbeb47e26b48e5b2f4942b1154ee5"}
Sep 10 12:45:06.108895 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.108797    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"
Sep 10 12:45:06.110318 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.110293    2563 generic.go:356] "Generic (PLEG): container finished" podID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerID="f9c4597fa2228e20cace0767699d8919caef6639beb574f15c5a1afb76f78c03" exitCode=0
Sep 10 12:45:06.110461 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.110340    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"
Sep 10 12:45:06.110461 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.110375    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" event={"ID":"b3667f65-20e0-4574-9440-89862bfe9d5c","Type":"ContainerDied","Data":"f9c4597fa2228e20cace0767699d8919caef6639beb574f15c5a1afb76f78c03"}
Sep 10 12:45:06.110461 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.110422    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k" event={"ID":"b3667f65-20e0-4574-9440-89862bfe9d5c","Type":"ContainerDied","Data":"b7a9bbd93cba89f86dd4b259025c0f3f64d105bd9113819ce0bf3f2d126e3f5d"}
Sep 10 12:45:06.110461 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.110445    2563 scope.go:117] "RemoveContainer" containerID="f9c4597fa2228e20cace0767699d8919caef6639beb574f15c5a1afb76f78c03"
Sep 10 12:45:06.121573 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.121534    2563 scope.go:117] "RemoveContainer" containerID="b7fcfd079e16fcca5e3a91a47ce18e861e9ceadbda05d6f1c493c6931ed4ce09"
Sep 10 12:45:06.130386 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.130363    2563 scope.go:117] "RemoveContainer" containerID="f9c4597fa2228e20cace0767699d8919caef6639beb574f15c5a1afb76f78c03"
Sep 10 12:45:06.130702 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:45:06.130680    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c4597fa2228e20cace0767699d8919caef6639beb574f15c5a1afb76f78c03\": container with ID starting with f9c4597fa2228e20cace0767699d8919caef6639beb574f15c5a1afb76f78c03 not found: ID does not exist" containerID="f9c4597fa2228e20cace0767699d8919caef6639beb574f15c5a1afb76f78c03"
Sep 10 12:45:06.130777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.130712    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c4597fa2228e20cace0767699d8919caef6639beb574f15c5a1afb76f78c03"} err="failed to get container status \"f9c4597fa2228e20cace0767699d8919caef6639beb574f15c5a1afb76f78c03\": rpc error: code = NotFound desc = could not find container \"f9c4597fa2228e20cace0767699d8919caef6639beb574f15c5a1afb76f78c03\": container with ID starting with f9c4597fa2228e20cace0767699d8919caef6639beb574f15c5a1afb76f78c03 not found: ID does not exist"
Sep 10 12:45:06.130777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.130730    2563 scope.go:117] "RemoveContainer" containerID="b7fcfd079e16fcca5e3a91a47ce18e861e9ceadbda05d6f1c493c6931ed4ce09"
Sep 10 12:45:06.131007 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:45:06.130992    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7fcfd079e16fcca5e3a91a47ce18e861e9ceadbda05d6f1c493c6931ed4ce09\": container with ID starting with b7fcfd079e16fcca5e3a91a47ce18e861e9ceadbda05d6f1c493c6931ed4ce09 not found: ID does not exist" containerID="b7fcfd079e16fcca5e3a91a47ce18e861e9ceadbda05d6f1c493c6931ed4ce09"
Sep 10 12:45:06.131051 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.131012    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7fcfd079e16fcca5e3a91a47ce18e861e9ceadbda05d6f1c493c6931ed4ce09"} err="failed to get container status \"b7fcfd079e16fcca5e3a91a47ce18e861e9ceadbda05d6f1c493c6931ed4ce09\": rpc error: code = NotFound desc = could not find container \"b7fcfd079e16fcca5e3a91a47ce18e861e9ceadbda05d6f1c493c6931ed4ce09\": container with ID starting with b7fcfd079e16fcca5e3a91a47ce18e861e9ceadbda05d6f1c493c6931ed4ce09 not found: ID does not exist"
Sep 10 12:45:06.149801 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.149747    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26" podStartSLOduration=2.149731702 podStartE2EDuration="2.149731702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:06.149129803 +0000 UTC m=+880.074041295" watchObservedRunningTime="2026-09-10 12:45:06.149731702 +0000 UTC m=+880.074643191"
Sep 10 12:45:06.206086 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.206049    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"]
Sep 10 12:45:06.228281 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.228243    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c8b03106-openshift-default-66dc565f8-shr8k"]
Sep 10 12:45:06.708490 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:06.708446    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" path="/var/lib/kubelet/pods/b3667f65-20e0-4574-9440-89862bfe9d5c/volumes"
Sep 10 12:45:10.379716 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:10.379674    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:10.380093 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:10.379732    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:10.392476 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:10.392446    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:11.187280 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:11.187244    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:45:17.119791 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:17.119756    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"
Sep 10 12:45:18.211125 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.211047    2563 patch_prober.go:28] interesting pod/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:35306->10.134.0.53:15021: read: connection reset by peer" start-of-body=
Sep 10 12:45:18.211642 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.211131    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:35306->10.134.0.53:15021: read: connection reset by peer"
Sep 10 12:45:18.639164 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.639048    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"]
Sep 10 12:45:18.639489 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.639476    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerName="istio-proxy"
Sep 10 12:45:18.639532 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.639490    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerName="istio-proxy"
Sep 10 12:45:18.639532 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.639503    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerName="istio-proxy"
Sep 10 12:45:18.639532 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.639509    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerName="istio-proxy"
Sep 10 12:45:18.639650 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.639598    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerName="istio-proxy"
Sep 10 12:45:18.641875 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.641858    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.651093 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.650931    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-30919a\"/\"kube-root-ca.crt\""
Sep 10 12:45:18.651093 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.650947    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-30919a\"/\"default-dockercfg-8xhp5\""
Sep 10 12:45:18.651093 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.650955    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-30919a\"/\"simulator-w0-30919a-kserve-self-signed-certs\""
Sep 10 12:45:18.651093 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.650952    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-30919a\"/\"openshift-service-ca.crt\""
Sep 10 12:45:18.668951 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.668921    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"]
Sep 10 12:45:18.803585 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.803536    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-tmp-dir\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.803789 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.803689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7znw\" (UniqueName: \"kubernetes.io/projected/4e63d3a2-3e29-41d2-924b-e87f8700d601-kube-api-access-x7znw\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.803877 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.803856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-model-cache\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.803937 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.803891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-dshm\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.804011 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.803989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e63d3a2-3e29-41d2-924b-e87f8700d601-tls-certs\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.804093 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.804064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-home\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.905299 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.905202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7znw\" (UniqueName: \"kubernetes.io/projected/4e63d3a2-3e29-41d2-924b-e87f8700d601-kube-api-access-x7znw\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.905299 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.905248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-model-cache\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.905299 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.905276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-dshm\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.905621 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.905306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e63d3a2-3e29-41d2-924b-e87f8700d601-tls-certs\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.905621 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.905341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-home\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.905621 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.905420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-tmp-dir\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.905807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.905664    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-model-cache\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.905807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.905782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-home\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.905885 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.905807    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-tmp-dir\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.907566 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.907523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-dshm\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.907811 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.907793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e63d3a2-3e29-41d2-924b-e87f8700d601-tls-certs\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.937785 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.937752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7znw\" (UniqueName: \"kubernetes.io/projected/4e63d3a2-3e29-41d2-924b-e87f8700d601-kube-api-access-x7znw\") pod \"simulator-w0-30919a-kserve-555c48d75b-gr9zk\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") " pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:18.952699 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:18.952668    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:19.217959 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:19.217928    2563 generic.go:356] "Generic (PLEG): container finished" podID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerID="04eae6173d569a4cedea97b58cbe7ef62b12fc74265281368ef5604a5d5eea2b" exitCode=0
Sep 10 12:45:19.218361 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:19.218001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" event={"ID":"5d8e5d8b-66af-46b4-92bd-094288d80ba9","Type":"ContainerDied","Data":"04eae6173d569a4cedea97b58cbe7ef62b12fc74265281368ef5604a5d5eea2b"}
Sep 10 12:45:19.218452 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:19.218438    2563 scope.go:117] "RemoveContainer" containerID="04eae6173d569a4cedea97b58cbe7ef62b12fc74265281368ef5604a5d5eea2b"
Sep 10 12:45:19.219731 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:19.219691    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:45:19.219854 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:19.219777    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:45:19.219854 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:19.219827    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:45:19.250543 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:19.250511    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"]
Sep 10 12:45:19.254903 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:45:19.254864    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e63d3a2_3e29_41d2_924b_e87f8700d601.slice/crio-a48ba596d72d840fd6aa17fcda3e9b77b78d1f9ac63d220e214a9feef93f3f46 WatchSource:0}: Error finding container a48ba596d72d840fd6aa17fcda3e9b77b78d1f9ac63d220e214a9feef93f3f46: Status 404 returned error can't find the container with id a48ba596d72d840fd6aa17fcda3e9b77b78d1f9ac63d220e214a9feef93f3f46
Sep 10 12:45:20.226451 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:20.226415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" event={"ID":"5d8e5d8b-66af-46b4-92bd-094288d80ba9","Type":"ContainerStarted","Data":"25a33b6df1c414258d4838a54cd16f59701ed5801cbc40a5cf96a852d29255ec"}
Sep 10 12:45:20.228059 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:20.228034    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk" event={"ID":"4e63d3a2-3e29-41d2-924b-e87f8700d601","Type":"ContainerStarted","Data":"3c63d510cb9791462366ee0c50db4c9eab7815e1a57cd75a94703eba16153d03"}
Sep 10 12:45:20.228174 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:20.228069    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk" event={"ID":"4e63d3a2-3e29-41d2-924b-e87f8700d601","Type":"ContainerStarted","Data":"a48ba596d72d840fd6aa17fcda3e9b77b78d1f9ac63d220e214a9feef93f3f46"}
Sep 10 12:45:20.311625 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:20.311590    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:20.311948 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:20.311923    2563 patch_prober.go:28] interesting pod/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 10 12:45:20.312003 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:20.311983    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 10 12:45:20.320661 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:20.320616    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk" podStartSLOduration=2.32060096 podStartE2EDuration="2.32060096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:20.320426426 +0000 UTC m=+894.245337915" watchObservedRunningTime="2026-09-10 12:45:20.32060096 +0000 UTC m=+894.245512448"
Sep 10 12:45:21.207263 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:21.207231    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:45:21.311903 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:21.311870    2563 patch_prober.go:28] interesting pod/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 10 12:45:21.312334 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:21.312306    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 10 12:45:22.311691 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:22.311654    2563 patch_prober.go:28] interesting pod/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 10 12:45:22.311893 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:22.311716    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 10 12:45:23.311485 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:23.311439    2563 patch_prober.go:28] interesting pod/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 10 12:45:23.311887 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:23.311533    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 10 12:45:24.312114 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:24.312079    2563 patch_prober.go:28] interesting pod/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 10 12:45:24.312504 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:24.312146    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 10 12:45:25.312169 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:25.312127    2563 patch_prober.go:28] interesting pod/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 10 12:45:25.312643 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:25.312186    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 10 12:45:26.315667 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:26.315637    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:26.316076 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:26.315925    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:26.316672 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:26.316653    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:27.308625 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.308594    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"]
Sep 10 12:45:27.309304 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.309284    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3667f65-20e0-4574-9440-89862bfe9d5c" containerName="istio-proxy"
Sep 10 12:45:27.313883 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.313861    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.318526 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.318506    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5abff0\"/\"default-dockercfg-lrd9p\""
Sep 10 12:45:27.319434 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.319410    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5abff0\"/\"openshift-service-ca.crt\""
Sep 10 12:45:27.319567 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.319445    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-5abff0\"/\"kube-root-ca.crt\""
Sep 10 12:45:27.319567 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.319472    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-5abff0\"/\"autores-model-5abff0-kserve-self-signed-certs\""
Sep 10 12:45:27.326262 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.326233    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"]
Sep 10 12:45:27.398168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.398129    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43c940d-8730-4a21-a026-4bdd07ba922c-tls-certs\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.398341 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.398181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-dshm\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.398341 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.398241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-model-cache\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.398341 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.398304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-home\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.398341 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.398325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28tlv\" (UniqueName: \"kubernetes.io/projected/f43c940d-8730-4a21-a026-4bdd07ba922c-kube-api-access-28tlv\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.398478 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.398404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-tmp-dir\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.498939 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.498900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43c940d-8730-4a21-a026-4bdd07ba922c-tls-certs\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.499148 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.498963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-dshm\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.499148 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.498988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-model-cache\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.499148 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.499032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-home\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.499148 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.499052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28tlv\" (UniqueName: \"kubernetes.io/projected/f43c940d-8730-4a21-a026-4bdd07ba922c-kube-api-access-28tlv\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.499148 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.499111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-tmp-dir\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.499478 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.499453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-home\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.499571 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.499532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-model-cache\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.499636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.499532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-tmp-dir\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.501394 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.501373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-dshm\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.501527 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.501508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43c940d-8730-4a21-a026-4bdd07ba922c-tls-certs\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.509493 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.509474    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28tlv\" (UniqueName: \"kubernetes.io/projected/f43c940d-8730-4a21-a026-4bdd07ba922c-kube-api-access-28tlv\") pod \"autores-model-5abff0-kserve-56cd98f87f-t9vgc\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") " pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.623955 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.623857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:27.918171 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:27.918142    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"]
Sep 10 12:45:27.920912 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:45:27.920884    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf43c940d_8730_4a21_a026_4bdd07ba922c.slice/crio-e98388dae015a41a9cfe03795e5f1f8a54b24efcfc234426c85cf92100538291 WatchSource:0}: Error finding container e98388dae015a41a9cfe03795e5f1f8a54b24efcfc234426c85cf92100538291: Status 404 returned error can't find the container with id e98388dae015a41a9cfe03795e5f1f8a54b24efcfc234426c85cf92100538291
Sep 10 12:45:28.283763 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:28.283670    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc" event={"ID":"f43c940d-8730-4a21-a026-4bdd07ba922c","Type":"ContainerStarted","Data":"c3b022ef97e0b12f6fbc65cb25bcfe8c8dbfb270abe41a3c3263a984e414a663"}
Sep 10 12:45:28.283763 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:28.283716    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc" event={"ID":"f43c940d-8730-4a21-a026-4bdd07ba922c","Type":"ContainerStarted","Data":"e98388dae015a41a9cfe03795e5f1f8a54b24efcfc234426c85cf92100538291"}
Sep 10 12:45:28.306368 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:28.306325    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc" podStartSLOduration=1.306309993 podStartE2EDuration="1.306309993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:28.305642153 +0000 UTC m=+902.230553661" watchObservedRunningTime="2026-09-10 12:45:28.306309993 +0000 UTC m=+902.231221530"
Sep 10 12:45:28.953102 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:28.953063    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:28.953495 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:28.953120    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:28.966450 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:28.966417    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:29.301123 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:29.301037    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:45:35.417887 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:35.417848    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"]
Sep 10 12:45:35.418279 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:35.418102    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26" podUID="c1226061-dc18-41e0-84fd-96e4917eeb9c" containerName="payload-processing" containerID="cri-o://51d7ebabce49ce3c2d676c7572866d935c3aecdf72ba5652445ea1ffc15f4ddd" gracePeriod=30
Sep 10 12:45:35.713174 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:35.713141    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"
Sep 10 12:45:35.777909 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:35.777872    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1226061-dc18-41e0-84fd-96e4917eeb9c-plugins-config-volume\") pod \"c1226061-dc18-41e0-84fd-96e4917eeb9c\" (UID: \"c1226061-dc18-41e0-84fd-96e4917eeb9c\") "
Sep 10 12:45:35.778117 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:35.777943    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx8sr\" (UniqueName: \"kubernetes.io/projected/c1226061-dc18-41e0-84fd-96e4917eeb9c-kube-api-access-qx8sr\") pod \"c1226061-dc18-41e0-84fd-96e4917eeb9c\" (UID: \"c1226061-dc18-41e0-84fd-96e4917eeb9c\") "
Sep 10 12:45:35.778307 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:35.778278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1226061-dc18-41e0-84fd-96e4917eeb9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1226061-dc18-41e0-84fd-96e4917eeb9c" (UID: "c1226061-dc18-41e0-84fd-96e4917eeb9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:45:35.780139 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:35.780102    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1226061-dc18-41e0-84fd-96e4917eeb9c-kube-api-access-qx8sr" (OuterVolumeSpecName: "kube-api-access-qx8sr") pod "c1226061-dc18-41e0-84fd-96e4917eeb9c" (UID: "c1226061-dc18-41e0-84fd-96e4917eeb9c"). InnerVolumeSpecName "kube-api-access-qx8sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:45:35.835860 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:35.835825    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"]
Sep 10 12:45:35.836088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:35.836064    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy" containerID="cri-o://25a33b6df1c414258d4838a54cd16f59701ed5801cbc40a5cf96a852d29255ec" gracePeriod=30
Sep 10 12:45:35.879402 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:35.879362    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1226061-dc18-41e0-84fd-96e4917eeb9c-plugins-config-volume\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:35.879402 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:35.879395    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx8sr\" (UniqueName: \"kubernetes.io/projected/c1226061-dc18-41e0-84fd-96e4917eeb9c-kube-api-access-qx8sr\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:36.334282 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.334244    2563 generic.go:356] "Generic (PLEG): container finished" podID="c1226061-dc18-41e0-84fd-96e4917eeb9c" containerID="51d7ebabce49ce3c2d676c7572866d935c3aecdf72ba5652445ea1ffc15f4ddd" exitCode=0
Sep 10 12:45:36.334470 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.334307    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"
Sep 10 12:45:36.334470 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.334331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26" event={"ID":"c1226061-dc18-41e0-84fd-96e4917eeb9c","Type":"ContainerDied","Data":"51d7ebabce49ce3c2d676c7572866d935c3aecdf72ba5652445ea1ffc15f4ddd"}
Sep 10 12:45:36.334470 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.334405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26" event={"ID":"c1226061-dc18-41e0-84fd-96e4917eeb9c","Type":"ContainerDied","Data":"7b372d837ec337f1fbc3a108080952485e9fbeb47e26b48e5b2f4942b1154ee5"}
Sep 10 12:45:36.334470 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.334429    2563 scope.go:117] "RemoveContainer" containerID="51d7ebabce49ce3c2d676c7572866d935c3aecdf72ba5652445ea1ffc15f4ddd"
Sep 10 12:45:36.343726 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.343630    2563 scope.go:117] "RemoveContainer" containerID="51d7ebabce49ce3c2d676c7572866d935c3aecdf72ba5652445ea1ffc15f4ddd"
Sep 10 12:45:36.343942 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:45:36.343923    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d7ebabce49ce3c2d676c7572866d935c3aecdf72ba5652445ea1ffc15f4ddd\": container with ID starting with 51d7ebabce49ce3c2d676c7572866d935c3aecdf72ba5652445ea1ffc15f4ddd not found: ID does not exist" containerID="51d7ebabce49ce3c2d676c7572866d935c3aecdf72ba5652445ea1ffc15f4ddd"
Sep 10 12:45:36.343999 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.343947    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d7ebabce49ce3c2d676c7572866d935c3aecdf72ba5652445ea1ffc15f4ddd"} err="failed to get container status \"51d7ebabce49ce3c2d676c7572866d935c3aecdf72ba5652445ea1ffc15f4ddd\": rpc error: code = NotFound desc = could not find container \"51d7ebabce49ce3c2d676c7572866d935c3aecdf72ba5652445ea1ffc15f4ddd\": container with ID starting with 51d7ebabce49ce3c2d676c7572866d935c3aecdf72ba5652445ea1ffc15f4ddd not found: ID does not exist"
Sep 10 12:45:36.393898 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.393863    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"]
Sep 10 12:45:36.400450 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.400425    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-760a0d05-6d9c956654-n7g26"]
Sep 10 12:45:36.708336 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.708292    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1226061-dc18-41e0-84fd-96e4917eeb9c" path="/var/lib/kubelet/pods/c1226061-dc18-41e0-84fd-96e4917eeb9c/volumes"
Sep 10 12:45:36.721433 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.721402    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"]
Sep 10 12:45:36.721985 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.721964    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1226061-dc18-41e0-84fd-96e4917eeb9c" containerName="payload-processing"
Sep 10 12:45:36.721985 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.721987    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1226061-dc18-41e0-84fd-96e4917eeb9c" containerName="payload-processing"
Sep 10 12:45:36.722147 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.722133    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1226061-dc18-41e0-84fd-96e4917eeb9c" containerName="payload-processing"
Sep 10 12:45:36.726947 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.726927    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.733037 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.733017    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d7fcd5f6-openshift-default-dockercfg-bvp6k\""
Sep 10 12:45:36.740890 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.740867    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"]
Sep 10 12:45:36.788904 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.788872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-token\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.789072 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.788927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-workload-socket\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.789072 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.788955    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istiod-ca-cert\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.789072 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.788981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-data\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.789072 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.789028    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-workload-certs\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.789072 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.789060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-podinfo\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.789238 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.789097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-credential-socket\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.789238 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.789151    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg7g7\" (UniqueName: \"kubernetes.io/projected/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-kube-api-access-kg7g7\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.789238 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.789204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-envoy\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.890565 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.890515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-token\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.890752 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.890620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-workload-socket\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.890752 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.890653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istiod-ca-cert\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.890842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.890737    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-data\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.890842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.890802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-workload-certs\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.890842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.890823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-podinfo\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.890975 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.890866    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-credential-socket\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.890975 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.890946    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg7g7\" (UniqueName: \"kubernetes.io/projected/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-kube-api-access-kg7g7\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.891077 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.890988    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-envoy\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.891209 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.891163    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-workload-socket\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.891209 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.891201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-workload-certs\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.891509 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.891478    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istiod-ca-cert\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.891635 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.891524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-data\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.891700 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.891661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-credential-socket\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.893279 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.893255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-envoy\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.893504 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.893488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-podinfo\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.900353 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.900328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-token\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:36.900803 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:36.900785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg7g7\" (UniqueName: \"kubernetes.io/projected/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-kube-api-access-kg7g7\") pod \"e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") " pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:37.039638 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:37.039530    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:37.328450 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:37.328423    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"]
Sep 10 12:45:37.330352 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:45:37.330325    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687b86c6_b1c2_4c4d_9909_53a8d7fd3a62.slice/crio-18d1e94a8e57231285a05af5eac85a5218222499316efb3ca9c5a77a63c991e4 WatchSource:0}: Error finding container 18d1e94a8e57231285a05af5eac85a5218222499316efb3ca9c5a77a63c991e4: Status 404 returned error can't find the container with id 18d1e94a8e57231285a05af5eac85a5218222499316efb3ca9c5a77a63c991e4
Sep 10 12:45:37.332809 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:37.332775    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:45:37.332891 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:37.332848    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:45:37.332891 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:37.332882    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:45:37.342880 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:37.342850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" event={"ID":"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62","Type":"ContainerStarted","Data":"18d1e94a8e57231285a05af5eac85a5218222499316efb3ca9c5a77a63c991e4"}
Sep 10 12:45:37.624034 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:37.623951    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:37.624034 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:37.624001    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:37.637355 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:37.637327    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:37.725635 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:37.725602    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:45:38.352576 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:38.352510    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" event={"ID":"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62","Type":"ContainerStarted","Data":"09a2672a40579098d44f4528f414bf75a017b56def7b427c5808692553bc8091"}
Sep 10 12:45:38.364674 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:38.364641    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:45:38.390900 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:38.390852    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" podStartSLOduration=2.390836247 podStartE2EDuration="2.390836247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:38.388925629 +0000 UTC m=+912.313837118" watchObservedRunningTime="2026-09-10 12:45:38.390836247 +0000 UTC m=+912.315747735"
Sep 10 12:45:39.039960 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:39.039917    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:39.045435 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:39.045411    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:39.359215 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:39.359105    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:39.360322 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:39.360267    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:45:41.108731 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.108651    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:41.240103 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.240067    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-credential-socket\") pod \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") "
Sep 10 12:45:41.240315 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.240116    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-data\") pod \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") "
Sep 10 12:45:41.240315 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.240187    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-workload-socket\") pod \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") "
Sep 10 12:45:41.240315 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.240251    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-podinfo\") pod \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") "
Sep 10 12:45:41.240315 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.240283    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-token\") pod \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") "
Sep 10 12:45:41.240315 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.240309    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv46j\" (UniqueName: \"kubernetes.io/projected/5d8e5d8b-66af-46b4-92bd-094288d80ba9-kube-api-access-sv46j\") pod \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") "
Sep 10 12:45:41.240596 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.240337    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istiod-ca-cert\") pod \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") "
Sep 10 12:45:41.240596 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.240373    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-workload-certs\") pod \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") "
Sep 10 12:45:41.240596 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.240402    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d8e5d8b-66af-46b4-92bd-094288d80ba9" (UID: "5d8e5d8b-66af-46b4-92bd-094288d80ba9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:45:41.240596 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.240416    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-envoy\") pod \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\" (UID: \"5d8e5d8b-66af-46b4-92bd-094288d80ba9\") "
Sep 10 12:45:41.241188 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.240919    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-credential-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:41.241188 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.241057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d8e5d8b-66af-46b4-92bd-094288d80ba9" (UID: "5d8e5d8b-66af-46b4-92bd-094288d80ba9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:45:41.241385 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.241270    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d8e5d8b-66af-46b4-92bd-094288d80ba9" (UID: "5d8e5d8b-66af-46b4-92bd-094288d80ba9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:45:41.241385 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.241344    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d8e5d8b-66af-46b4-92bd-094288d80ba9" (UID: "5d8e5d8b-66af-46b4-92bd-094288d80ba9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:45:41.241684 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.241639    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d8e5d8b-66af-46b4-92bd-094288d80ba9" (UID: "5d8e5d8b-66af-46b4-92bd-094288d80ba9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:45:41.243093 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.243054    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d8e5d8b-66af-46b4-92bd-094288d80ba9" (UID: "5d8e5d8b-66af-46b4-92bd-094288d80ba9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:45:41.243382 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.243357    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8e5d8b-66af-46b4-92bd-094288d80ba9-kube-api-access-sv46j" (OuterVolumeSpecName: "kube-api-access-sv46j") pod "5d8e5d8b-66af-46b4-92bd-094288d80ba9" (UID: "5d8e5d8b-66af-46b4-92bd-094288d80ba9"). InnerVolumeSpecName "kube-api-access-sv46j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:45:41.243452 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.243424    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d8e5d8b-66af-46b4-92bd-094288d80ba9" (UID: "5d8e5d8b-66af-46b4-92bd-094288d80ba9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:45:41.243510 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.243496    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d8e5d8b-66af-46b4-92bd-094288d80ba9" (UID: "5d8e5d8b-66af-46b4-92bd-094288d80ba9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:45:41.342108 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.342064    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-token\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:41.342108 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.342103    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv46j\" (UniqueName: \"kubernetes.io/projected/5d8e5d8b-66af-46b4-92bd-094288d80ba9-kube-api-access-sv46j\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:41.342108 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.342117    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istiod-ca-cert\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:41.342362 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.342128    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-workload-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:41.342362 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.342139    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-envoy\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:41.342362 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.342148    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-data\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:41.342362 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.342159    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8e5d8b-66af-46b4-92bd-094288d80ba9-workload-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:41.342362 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.342169    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8e5d8b-66af-46b4-92bd-094288d80ba9-istio-podinfo\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:45:41.374868 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.372090    2563 generic.go:356] "Generic (PLEG): container finished" podID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerID="25a33b6df1c414258d4838a54cd16f59701ed5801cbc40a5cf96a852d29255ec" exitCode=0
Sep 10 12:45:41.374868 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.372967    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"
Sep 10 12:45:41.375095 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.374879    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" event={"ID":"5d8e5d8b-66af-46b4-92bd-094288d80ba9","Type":"ContainerDied","Data":"25a33b6df1c414258d4838a54cd16f59701ed5801cbc40a5cf96a852d29255ec"}
Sep 10 12:45:41.375095 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.374944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm" event={"ID":"5d8e5d8b-66af-46b4-92bd-094288d80ba9","Type":"ContainerDied","Data":"d4ee10000733f69932a8699644310ddbcaacaa4a1bc133bba9d501a92556ec3d"}
Sep 10 12:45:41.375095 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.374978    2563 scope.go:117] "RemoveContainer" containerID="25a33b6df1c414258d4838a54cd16f59701ed5801cbc40a5cf96a852d29255ec"
Sep 10 12:45:41.385443 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.385420    2563 scope.go:117] "RemoveContainer" containerID="04eae6173d569a4cedea97b58cbe7ef62b12fc74265281368ef5604a5d5eea2b"
Sep 10 12:45:41.394133 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.394109    2563 scope.go:117] "RemoveContainer" containerID="25a33b6df1c414258d4838a54cd16f59701ed5801cbc40a5cf96a852d29255ec"
Sep 10 12:45:41.394453 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:45:41.394434    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a33b6df1c414258d4838a54cd16f59701ed5801cbc40a5cf96a852d29255ec\": container with ID starting with 25a33b6df1c414258d4838a54cd16f59701ed5801cbc40a5cf96a852d29255ec not found: ID does not exist" containerID="25a33b6df1c414258d4838a54cd16f59701ed5801cbc40a5cf96a852d29255ec"
Sep 10 12:45:41.394499 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.394464    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a33b6df1c414258d4838a54cd16f59701ed5801cbc40a5cf96a852d29255ec"} err="failed to get container status \"25a33b6df1c414258d4838a54cd16f59701ed5801cbc40a5cf96a852d29255ec\": rpc error: code = NotFound desc = could not find container \"25a33b6df1c414258d4838a54cd16f59701ed5801cbc40a5cf96a852d29255ec\": container with ID starting with 25a33b6df1c414258d4838a54cd16f59701ed5801cbc40a5cf96a852d29255ec not found: ID does not exist"
Sep 10 12:45:41.394499 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.394482    2563 scope.go:117] "RemoveContainer" containerID="04eae6173d569a4cedea97b58cbe7ef62b12fc74265281368ef5604a5d5eea2b"
Sep 10 12:45:41.394770 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:45:41.394756    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04eae6173d569a4cedea97b58cbe7ef62b12fc74265281368ef5604a5d5eea2b\": container with ID starting with 04eae6173d569a4cedea97b58cbe7ef62b12fc74265281368ef5604a5d5eea2b not found: ID does not exist" containerID="04eae6173d569a4cedea97b58cbe7ef62b12fc74265281368ef5604a5d5eea2b"
Sep 10 12:45:41.394819 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.394775    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04eae6173d569a4cedea97b58cbe7ef62b12fc74265281368ef5604a5d5eea2b"} err="failed to get container status \"04eae6173d569a4cedea97b58cbe7ef62b12fc74265281368ef5604a5d5eea2b\": rpc error: code = NotFound desc = could not find container \"04eae6173d569a4cedea97b58cbe7ef62b12fc74265281368ef5604a5d5eea2b\": container with ID starting with 04eae6173d569a4cedea97b58cbe7ef62b12fc74265281368ef5604a5d5eea2b not found: ID does not exist"
Sep 10 12:45:41.401956 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.401930    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"]
Sep 10 12:45:41.405932 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:41.405909    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-760a0d05-gw-openshift-default-5f968b8d9b-vpftm"]
Sep 10 12:45:42.705776 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:42.705734    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" path="/var/lib/kubelet/pods/5d8e5d8b-66af-46b4-92bd-094288d80ba9/volumes"
Sep 10 12:45:59.490440 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.490402    2563 generic.go:356] "Generic (PLEG): container finished" podID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerID="09a2672a40579098d44f4528f414bf75a017b56def7b427c5808692553bc8091" exitCode=0
Sep 10 12:45:59.490876 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.490479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" event={"ID":"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62","Type":"ContainerDied","Data":"09a2672a40579098d44f4528f414bf75a017b56def7b427c5808692553bc8091"}
Sep 10 12:45:59.491020 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.491004    2563 scope.go:117] "RemoveContainer" containerID="09a2672a40579098d44f4528f414bf75a017b56def7b427c5808692553bc8091"
Sep 10 12:45:59.492259 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.492224    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:45:59.492359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.492292    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:45:59.492359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.492324    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:45:59.922915 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.922882    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682"]
Sep 10 12:45:59.923519 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.923500    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy"
Sep 10 12:45:59.923592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.923522    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy"
Sep 10 12:45:59.923684 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.923668    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy"
Sep 10 12:45:59.926418 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.926400    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682"
Sep 10 12:45:59.929102 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.929082    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:45:59.929335 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.929321    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:45:59.938336 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:45:59.938309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682"]
Sep 10 12:46:00.042108 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:00.042068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f94d909-18cf-4fa1-9a70-3de166eb886a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5abff0-h5682\" (UID: \"8f94d909-18cf-4fa1-9a70-3de166eb886a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682"
Sep 10 12:46:00.143661 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:00.143624    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f94d909-18cf-4fa1-9a70-3de166eb886a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5abff0-h5682\" (UID: \"8f94d909-18cf-4fa1-9a70-3de166eb886a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682"
Sep 10 12:46:00.144393 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:00.144372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f94d909-18cf-4fa1-9a70-3de166eb886a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5abff0-h5682\" (UID: \"8f94d909-18cf-4fa1-9a70-3de166eb886a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682"
Sep 10 12:46:00.237856 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:00.237760    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682"
Sep 10 12:46:00.499911 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:00.499799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" event={"ID":"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62","Type":"ContainerStarted","Data":"d581ebaf5feca40108e5ecd847a1064147e90167d593b272375f0f5122e461c5"}
Sep 10 12:46:00.519625 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:00.519596    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682"]
Sep 10 12:46:00.521516 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:46:00.521488    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f94d909_18cf_4fa1_9a70_3de166eb886a.slice/crio-7fd747bae5ba2224a62d1e961f4ef4098d36bf5be276aa6b98a3fe2cf9a585bf WatchSource:0}: Error finding container 7fd747bae5ba2224a62d1e961f4ef4098d36bf5be276aa6b98a3fe2cf9a585bf: Status 404 returned error can't find the container with id 7fd747bae5ba2224a62d1e961f4ef4098d36bf5be276aa6b98a3fe2cf9a585bf
Sep 10 12:46:01.040384 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:01.040293    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:46:01.040675 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:01.040651    2563 patch_prober.go:28] interesting pod/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 12:46:01.040774 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:01.040709    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 12:46:01.507656 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:01.507623    2563 generic.go:356] "Generic (PLEG): container finished" podID="8f94d909-18cf-4fa1-9a70-3de166eb886a" containerID="b009b94bb03d72f1b044e727cf5c02dede5b4964eec9f114e82a5e25cce952b9" exitCode=0
Sep 10 12:46:01.508102 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:01.507711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682" event={"ID":"8f94d909-18cf-4fa1-9a70-3de166eb886a","Type":"ContainerDied","Data":"b009b94bb03d72f1b044e727cf5c02dede5b4964eec9f114e82a5e25cce952b9"}
Sep 10 12:46:01.508102 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:01.507762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682" event={"ID":"8f94d909-18cf-4fa1-9a70-3de166eb886a","Type":"ContainerStarted","Data":"7fd747bae5ba2224a62d1e961f4ef4098d36bf5be276aa6b98a3fe2cf9a585bf"}
Sep 10 12:46:02.040207 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:02.040175    2563 patch_prober.go:28] interesting pod/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 12:46:02.040359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:02.040233    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 12:46:02.696715 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:02.696689    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682"
Sep 10 12:46:02.878654 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:02.878563    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f94d909-18cf-4fa1-9a70-3de166eb886a-maas-api-service-ca\") pod \"8f94d909-18cf-4fa1-9a70-3de166eb886a\" (UID: \"8f94d909-18cf-4fa1-9a70-3de166eb886a\") "
Sep 10 12:46:02.878874 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:02.878850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f94d909-18cf-4fa1-9a70-3de166eb886a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f94d909-18cf-4fa1-9a70-3de166eb886a" (UID: "8f94d909-18cf-4fa1-9a70-3de166eb886a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:46:02.878974 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:02.878963    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f94d909-18cf-4fa1-9a70-3de166eb886a-maas-api-service-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:02.925117 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:02.925084    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:46:03.040369 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:03.040335    2563 patch_prober.go:28] interesting pod/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 12:46:03.040524 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:03.040390    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 12:46:03.521127 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:03.521096    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682"
Sep 10 12:46:03.521314 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:03.521141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682" event={"ID":"8f94d909-18cf-4fa1-9a70-3de166eb886a","Type":"ContainerDied","Data":"7fd747bae5ba2224a62d1e961f4ef4098d36bf5be276aa6b98a3fe2cf9a585bf"}
Sep 10 12:46:03.521314 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:03.521199    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fd747bae5ba2224a62d1e961f4ef4098d36bf5be276aa6b98a3fe2cf9a585bf"
Sep 10 12:46:04.040835 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:04.040796    2563 patch_prober.go:28] interesting pod/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 12:46:04.041201 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:04.040880    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 12:46:05.040102 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.040064    2563 patch_prober.go:28] interesting pod/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 12:46:05.040284 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.040125    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 12:46:05.554909 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.554873    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk"]
Sep 10 12:46:05.556159 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.556128    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f94d909-18cf-4fa1-9a70-3de166eb886a" containerName="revoke-keys"
Sep 10 12:46:05.556323 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.556309    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f94d909-18cf-4fa1-9a70-3de166eb886a" containerName="revoke-keys"
Sep 10 12:46:05.556428 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.556416    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy"
Sep 10 12:46:05.556513 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.556502    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy"
Sep 10 12:46:05.556772 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.556755    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f94d909-18cf-4fa1-9a70-3de166eb886a" containerName="revoke-keys"
Sep 10 12:46:05.556901 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.556887    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d8e5d8b-66af-46b4-92bd-094288d80ba9" containerName="istio-proxy"
Sep 10 12:46:05.560535 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.560515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk"
Sep 10 12:46:05.569362 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.569326    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:46:05.569648 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.569626    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:46:05.572484 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.572451    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk"]
Sep 10 12:46:05.605436 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.605401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3fafec-236d-4cf4-884b-d69a6681bec7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk\" (UID: \"3a3fafec-236d-4cf4-884b-d69a6681bec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk"
Sep 10 12:46:05.706867 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.706829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3fafec-236d-4cf4-884b-d69a6681bec7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk\" (UID: \"3a3fafec-236d-4cf4-884b-d69a6681bec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk"
Sep 10 12:46:05.707474 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.707452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3fafec-236d-4cf4-884b-d69a6681bec7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk\" (UID: \"3a3fafec-236d-4cf4-884b-d69a6681bec7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk"
Sep 10 12:46:05.889453 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:05.889357    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk"
Sep 10 12:46:06.040783 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:06.040744    2563 patch_prober.go:28] interesting pod/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Sep 10 12:46:06.040938 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:06.040812    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 10 12:46:06.172716 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:06.171010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk"]
Sep 10 12:46:06.174202 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:46:06.174159    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3fafec_236d_4cf4_884b_d69a6681bec7.slice/crio-91191daa125b19db830beda635037ac18100a2e400b8024ab225b45c78e0a78c WatchSource:0}: Error finding container 91191daa125b19db830beda635037ac18100a2e400b8024ab225b45c78e0a78c: Status 404 returned error can't find the container with id 91191daa125b19db830beda635037ac18100a2e400b8024ab225b45c78e0a78c
Sep 10 12:46:06.541504 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:06.541413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk" event={"ID":"3a3fafec-236d-4cf4-884b-d69a6681bec7","Type":"ContainerStarted","Data":"91191daa125b19db830beda635037ac18100a2e400b8024ab225b45c78e0a78c"}
Sep 10 12:46:07.163179 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:07.163098    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:46:07.163697 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:07.163643    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:46:07.164740 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:07.164720    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:46:07.550460 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:07.550422    2563 generic.go:356] "Generic (PLEG): container finished" podID="3a3fafec-236d-4cf4-884b-d69a6681bec7" containerID="cfbf3ff9b89ead52c7eb9210d0434254ca8a3035380031b0a8e2659b74c4452b" exitCode=0
Sep 10 12:46:07.550688 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:07.550502    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk" event={"ID":"3a3fafec-236d-4cf4-884b-d69a6681bec7","Type":"ContainerDied","Data":"cfbf3ff9b89ead52c7eb9210d0434254ca8a3035380031b0a8e2659b74c4452b"}
Sep 10 12:46:08.694991 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:08.694965    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk"
Sep 10 12:46:08.739084 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:08.739035    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3fafec-236d-4cf4-884b-d69a6681bec7-maas-api-service-ca\") pod \"3a3fafec-236d-4cf4-884b-d69a6681bec7\" (UID: \"3a3fafec-236d-4cf4-884b-d69a6681bec7\") "
Sep 10 12:46:08.739417 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:08.739390    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3fafec-236d-4cf4-884b-d69a6681bec7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a3fafec-236d-4cf4-884b-d69a6681bec7" (UID: "3a3fafec-236d-4cf4-884b-d69a6681bec7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:46:08.840317 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:08.840214    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3fafec-236d-4cf4-884b-d69a6681bec7-maas-api-service-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:09.565001 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:09.564964    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk"
Sep 10 12:46:09.565183 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:09.564999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk" event={"ID":"3a3fafec-236d-4cf4-884b-d69a6681bec7","Type":"ContainerDied","Data":"91191daa125b19db830beda635037ac18100a2e400b8024ab225b45c78e0a78c"}
Sep 10 12:46:09.565183 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:09.565050    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91191daa125b19db830beda635037ac18100a2e400b8024ab225b45c78e0a78c"
Sep 10 12:46:10.921234 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:10.921179    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5abff0-29817405-79lnd"]
Sep 10 12:46:10.926085 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:10.926059    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-5abff0-29817405-79lnd"]
Sep 10 12:46:11.297589 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:11.297488    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682"]
Sep 10 12:46:11.302196 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:11.302168    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5abff0-h5682"]
Sep 10 12:46:12.705715 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:12.705679    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f94d909-18cf-4fa1-9a70-3de166eb886a" path="/var/lib/kubelet/pods/8f94d909-18cf-4fa1-9a70-3de166eb886a/volumes"
Sep 10 12:46:12.706080 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:12.705986    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93b621bd-daf2-4dea-b31c-62b6c73fea8a" path="/var/lib/kubelet/pods/93b621bd-daf2-4dea-b31c-62b6c73fea8a/volumes"
Sep 10 12:46:12.706227 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:12.706213    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk"]
Sep 10 12:46:12.706262 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:12.706242    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d7fcd5f6-rczlk"]
Sep 10 12:46:13.848745 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:13.848708    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"]
Sep 10 12:46:13.849213 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:13.848980    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerName="istio-proxy" containerID="cri-o://d581ebaf5feca40108e5ecd847a1064147e90167d593b272375f0f5122e461c5" gracePeriod=30
Sep 10 12:46:14.705986 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:14.705941    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a3fafec-236d-4cf4-884b-d69a6681bec7" path="/var/lib/kubelet/pods/3a3fafec-236d-4cf4-884b-d69a6681bec7/volumes"
Sep 10 12:46:16.644647 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:16.644619    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"]
Sep 10 12:46:19.113714 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.113654    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:46:19.254778 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.254745    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-data\") pod \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") "
Sep 10 12:46:19.254975 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.254798    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-token\") pod \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") "
Sep 10 12:46:19.254975 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.254817    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-podinfo\") pod \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") "
Sep 10 12:46:19.254975 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.254858    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-workload-socket\") pod \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") "
Sep 10 12:46:19.254975 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.254890    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-workload-certs\") pod \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") "
Sep 10 12:46:19.254975 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.254940    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg7g7\" (UniqueName: \"kubernetes.io/projected/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-kube-api-access-kg7g7\") pod \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") "
Sep 10 12:46:19.255256 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.254996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-credential-socket\") pod \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") "
Sep 10 12:46:19.255256 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.255058    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-data" (OuterVolumeSpecName: "istio-data") pod "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" (UID: "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:46:19.255256 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.255072    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istiod-ca-cert\") pod \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") "
Sep 10 12:46:19.255256 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.255132    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-envoy\") pod \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\" (UID: \"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62\") "
Sep 10 12:46:19.255467 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.255368    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" (UID: "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:46:19.255467 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.255369    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" (UID: "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:46:19.255665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.255508    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" (UID: "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:46:19.255665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.255610    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-credential-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:19.255665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.255627    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istiod-ca-cert\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:19.255665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.255635    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-data\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:19.255665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.255643    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-workload-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:19.255927 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.255724    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" (UID: "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:46:19.257338 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.257313    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-token" (OuterVolumeSpecName: "istio-token") pod "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" (UID: "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:46:19.257431 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.257409    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" (UID: "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:46:19.257481 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.257429    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" (UID: "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:46:19.257523 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.257502    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-kube-api-access-kg7g7" (OuterVolumeSpecName: "kube-api-access-kg7g7") pod "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" (UID: "687b86c6-b1c2-4c4d-9909-53a8d7fd3a62"). InnerVolumeSpecName "kube-api-access-kg7g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:46:19.357105 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.357069    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-workload-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:19.357105 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.357100    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg7g7\" (UniqueName: \"kubernetes.io/projected/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-kube-api-access-kg7g7\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:19.357105 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.357110    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-envoy\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:19.357320 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.357118    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-token\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:19.357320 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.357126    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62-istio-podinfo\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:19.630640 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.630608    2563 generic.go:356] "Generic (PLEG): container finished" podID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerID="d581ebaf5feca40108e5ecd847a1064147e90167d593b272375f0f5122e461c5" exitCode=0
Sep 10 12:46:19.630835 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.630660    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"
Sep 10 12:46:19.630835 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.630694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" event={"ID":"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62","Type":"ContainerDied","Data":"d581ebaf5feca40108e5ecd847a1064147e90167d593b272375f0f5122e461c5"}
Sep 10 12:46:19.630835 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.630754    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q" event={"ID":"687b86c6-b1c2-4c4d-9909-53a8d7fd3a62","Type":"ContainerDied","Data":"18d1e94a8e57231285a05af5eac85a5218222499316efb3ca9c5a77a63c991e4"}
Sep 10 12:46:19.630835 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.630779    2563 scope.go:117] "RemoveContainer" containerID="d581ebaf5feca40108e5ecd847a1064147e90167d593b272375f0f5122e461c5"
Sep 10 12:46:19.640305 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.640285    2563 scope.go:117] "RemoveContainer" containerID="09a2672a40579098d44f4528f414bf75a017b56def7b427c5808692553bc8091"
Sep 10 12:46:19.648940 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.648916    2563 scope.go:117] "RemoveContainer" containerID="d581ebaf5feca40108e5ecd847a1064147e90167d593b272375f0f5122e461c5"
Sep 10 12:46:19.649224 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:46:19.649204    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d581ebaf5feca40108e5ecd847a1064147e90167d593b272375f0f5122e461c5\": container with ID starting with d581ebaf5feca40108e5ecd847a1064147e90167d593b272375f0f5122e461c5 not found: ID does not exist" containerID="d581ebaf5feca40108e5ecd847a1064147e90167d593b272375f0f5122e461c5"
Sep 10 12:46:19.649297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.649233    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d581ebaf5feca40108e5ecd847a1064147e90167d593b272375f0f5122e461c5"} err="failed to get container status \"d581ebaf5feca40108e5ecd847a1064147e90167d593b272375f0f5122e461c5\": rpc error: code = NotFound desc = could not find container \"d581ebaf5feca40108e5ecd847a1064147e90167d593b272375f0f5122e461c5\": container with ID starting with d581ebaf5feca40108e5ecd847a1064147e90167d593b272375f0f5122e461c5 not found: ID does not exist"
Sep 10 12:46:19.649297 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.649251    2563 scope.go:117] "RemoveContainer" containerID="09a2672a40579098d44f4528f414bf75a017b56def7b427c5808692553bc8091"
Sep 10 12:46:19.649501 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:46:19.649484    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a2672a40579098d44f4528f414bf75a017b56def7b427c5808692553bc8091\": container with ID starting with 09a2672a40579098d44f4528f414bf75a017b56def7b427c5808692553bc8091 not found: ID does not exist" containerID="09a2672a40579098d44f4528f414bf75a017b56def7b427c5808692553bc8091"
Sep 10 12:46:19.649591 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.649504    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a2672a40579098d44f4528f414bf75a017b56def7b427c5808692553bc8091"} err="failed to get container status \"09a2672a40579098d44f4528f414bf75a017b56def7b427c5808692553bc8091\": rpc error: code = NotFound desc = could not find container \"09a2672a40579098d44f4528f414bf75a017b56def7b427c5808692553bc8091\": container with ID starting with 09a2672a40579098d44f4528f414bf75a017b56def7b427c5808692553bc8091 not found: ID does not exist"
Sep 10 12:46:19.654176 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.654156    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"]
Sep 10 12:46:19.660241 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:19.660220    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d7fcd5f6-openshift-default-68bbcff7f4-gt89q"]
Sep 10 12:46:20.240343 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:20.240313    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:46:20.707249 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:20.707216    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" path="/var/lib/kubelet/pods/687b86c6-b1c2-4c4d-9909-53a8d7fd3a62/volumes"
Sep 10 12:46:31.664824 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:31.664752    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc" podUID="f43c940d-8730-4a21-a026-4bdd07ba922c" containerName="main" containerID="cri-o://c3b022ef97e0b12f6fbc65cb25bcfe8c8dbfb270abe41a3c3263a984e414a663" gracePeriod=45
Sep 10 12:46:31.925473 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:31.925445    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:46:32.087408 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.087366    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-home\") pod \"f43c940d-8730-4a21-a026-4bdd07ba922c\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") "
Sep 10 12:46:32.087601 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.087445    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43c940d-8730-4a21-a026-4bdd07ba922c-tls-certs\") pod \"f43c940d-8730-4a21-a026-4bdd07ba922c\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") "
Sep 10 12:46:32.087601 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.087477    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-model-cache\") pod \"f43c940d-8730-4a21-a026-4bdd07ba922c\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") "
Sep 10 12:46:32.087601 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.087498    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-dshm\") pod \"f43c940d-8730-4a21-a026-4bdd07ba922c\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") "
Sep 10 12:46:32.087601 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.087529    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-tmp-dir\") pod \"f43c940d-8730-4a21-a026-4bdd07ba922c\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") "
Sep 10 12:46:32.087827 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.087652    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28tlv\" (UniqueName: \"kubernetes.io/projected/f43c940d-8730-4a21-a026-4bdd07ba922c-kube-api-access-28tlv\") pod \"f43c940d-8730-4a21-a026-4bdd07ba922c\" (UID: \"f43c940d-8730-4a21-a026-4bdd07ba922c\") "
Sep 10 12:46:32.087827 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.087729    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-home" (OuterVolumeSpecName: "home") pod "f43c940d-8730-4a21-a026-4bdd07ba922c" (UID: "f43c940d-8730-4a21-a026-4bdd07ba922c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:46:32.087827 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.087766    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-model-cache" (OuterVolumeSpecName: "model-cache") pod "f43c940d-8730-4a21-a026-4bdd07ba922c" (UID: "f43c940d-8730-4a21-a026-4bdd07ba922c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:46:32.087974 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.087871    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f43c940d-8730-4a21-a026-4bdd07ba922c" (UID: "f43c940d-8730-4a21-a026-4bdd07ba922c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:46:32.088040 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.088021    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:32.088080 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.088046    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:32.088080 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.088058    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:32.089781 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.089757    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-dshm" (OuterVolumeSpecName: "dshm") pod "f43c940d-8730-4a21-a026-4bdd07ba922c" (UID: "f43c940d-8730-4a21-a026-4bdd07ba922c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:46:32.089888 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.089871    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43c940d-8730-4a21-a026-4bdd07ba922c-kube-api-access-28tlv" (OuterVolumeSpecName: "kube-api-access-28tlv") pod "f43c940d-8730-4a21-a026-4bdd07ba922c" (UID: "f43c940d-8730-4a21-a026-4bdd07ba922c"). InnerVolumeSpecName "kube-api-access-28tlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:46:32.089935 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.089893    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43c940d-8730-4a21-a026-4bdd07ba922c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f43c940d-8730-4a21-a026-4bdd07ba922c" (UID: "f43c940d-8730-4a21-a026-4bdd07ba922c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:46:32.189333 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.189243    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28tlv\" (UniqueName: \"kubernetes.io/projected/f43c940d-8730-4a21-a026-4bdd07ba922c-kube-api-access-28tlv\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:32.189333 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.189278    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43c940d-8730-4a21-a026-4bdd07ba922c-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:32.189333 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.189288    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43c940d-8730-4a21-a026-4bdd07ba922c-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:32.709584 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.709522    2563 generic.go:356] "Generic (PLEG): container finished" podID="f43c940d-8730-4a21-a026-4bdd07ba922c" containerID="c3b022ef97e0b12f6fbc65cb25bcfe8c8dbfb270abe41a3c3263a984e414a663" exitCode=0
Sep 10 12:46:32.709584 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.709586    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"
Sep 10 12:46:32.710087 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.709589    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc" event={"ID":"f43c940d-8730-4a21-a026-4bdd07ba922c","Type":"ContainerDied","Data":"c3b022ef97e0b12f6fbc65cb25bcfe8c8dbfb270abe41a3c3263a984e414a663"}
Sep 10 12:46:32.710087 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.709632    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc" event={"ID":"f43c940d-8730-4a21-a026-4bdd07ba922c","Type":"ContainerDied","Data":"e98388dae015a41a9cfe03795e5f1f8a54b24efcfc234426c85cf92100538291"}
Sep 10 12:46:32.710087 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.709654    2563 scope.go:117] "RemoveContainer" containerID="c3b022ef97e0b12f6fbc65cb25bcfe8c8dbfb270abe41a3c3263a984e414a663"
Sep 10 12:46:32.719140 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.719123    2563 scope.go:117] "RemoveContainer" containerID="c3b022ef97e0b12f6fbc65cb25bcfe8c8dbfb270abe41a3c3263a984e414a663"
Sep 10 12:46:32.719426 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:46:32.719405    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b022ef97e0b12f6fbc65cb25bcfe8c8dbfb270abe41a3c3263a984e414a663\": container with ID starting with c3b022ef97e0b12f6fbc65cb25bcfe8c8dbfb270abe41a3c3263a984e414a663 not found: ID does not exist" containerID="c3b022ef97e0b12f6fbc65cb25bcfe8c8dbfb270abe41a3c3263a984e414a663"
Sep 10 12:46:32.719475 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.719445    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b022ef97e0b12f6fbc65cb25bcfe8c8dbfb270abe41a3c3263a984e414a663"} err="failed to get container status \"c3b022ef97e0b12f6fbc65cb25bcfe8c8dbfb270abe41a3c3263a984e414a663\": rpc error: code = NotFound desc = could not find container \"c3b022ef97e0b12f6fbc65cb25bcfe8c8dbfb270abe41a3c3263a984e414a663\": container with ID starting with c3b022ef97e0b12f6fbc65cb25bcfe8c8dbfb270abe41a3c3263a984e414a663 not found: ID does not exist"
Sep 10 12:46:32.732298 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.732272    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"]
Sep 10 12:46:32.737984 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:32.737959    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-5abff0/autores-model-5abff0-kserve-56cd98f87f-t9vgc"]
Sep 10 12:46:34.706158 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:34.706122    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f43c940d-8730-4a21-a026-4bdd07ba922c" path="/var/lib/kubelet/pods/f43c940d-8730-4a21-a026-4bdd07ba922c/volumes"
Sep 10 12:46:38.768382 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:38.768344    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:46:42.205467 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.205431    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx"]
Sep 10 12:46:42.206137 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.206115    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a3fafec-236d-4cf4-884b-d69a6681bec7" containerName="revoke-keys"
Sep 10 12:46:42.206231 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.206139    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a3fafec-236d-4cf4-884b-d69a6681bec7" containerName="revoke-keys"
Sep 10 12:46:42.206231 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.206173    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerName="istio-proxy"
Sep 10 12:46:42.206231 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.206182    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerName="istio-proxy"
Sep 10 12:46:42.206231 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.206194    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerName="istio-proxy"
Sep 10 12:46:42.206231 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.206202    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerName="istio-proxy"
Sep 10 12:46:42.206231 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.206215    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f43c940d-8730-4a21-a026-4bdd07ba922c" containerName="main"
Sep 10 12:46:42.206231 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.206223    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f43c940d-8730-4a21-a026-4bdd07ba922c" containerName="main"
Sep 10 12:46:42.206613 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.206351    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerName="istio-proxy"
Sep 10 12:46:42.206613 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.206365    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f43c940d-8730-4a21-a026-4bdd07ba922c" containerName="main"
Sep 10 12:46:42.206613 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.206378    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="687b86c6-b1c2-4c4d-9909-53a8d7fd3a62" containerName="istio-proxy"
Sep 10 12:46:42.206613 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.206389    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a3fafec-236d-4cf4-884b-d69a6681bec7" containerName="revoke-keys"
Sep 10 12:46:42.209931 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.209910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx"
Sep 10 12:46:42.213612 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.213580    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:46:42.213724 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.213673    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:46:42.223319 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.223294    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx"]
Sep 10 12:46:42.291275 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.291241    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11292fdd-e203-4393-befd-a2b9a005cc3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx\" (UID: \"11292fdd-e203-4393-befd-a2b9a005cc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx"
Sep 10 12:46:42.392681 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.392636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11292fdd-e203-4393-befd-a2b9a005cc3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx\" (UID: \"11292fdd-e203-4393-befd-a2b9a005cc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx"
Sep 10 12:46:42.393284 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.393262    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11292fdd-e203-4393-befd-a2b9a005cc3c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx\" (UID: \"11292fdd-e203-4393-befd-a2b9a005cc3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx"
Sep 10 12:46:42.521456 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.521371    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx"
Sep 10 12:46:42.797694 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:42.797666    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx"]
Sep 10 12:46:42.800056 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:46:42.800030    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11292fdd_e203_4393_befd_a2b9a005cc3c.slice/crio-dabcd669d70111c04f30747768b9a9539771f2089faa6a5a8ab4b5761341dbf4 WatchSource:0}: Error finding container dabcd669d70111c04f30747768b9a9539771f2089faa6a5a8ab4b5761341dbf4: Status 404 returned error can't find the container with id dabcd669d70111c04f30747768b9a9539771f2089faa6a5a8ab4b5761341dbf4
Sep 10 12:46:43.783607 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:43.783507    2563 generic.go:356] "Generic (PLEG): container finished" podID="11292fdd-e203-4393-befd-a2b9a005cc3c" containerID="b583e6a531c73e87aa6415268f3a1d388dbd320722d63de823583db20218a97a" exitCode=0
Sep 10 12:46:43.783607 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:43.783593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx" event={"ID":"11292fdd-e203-4393-befd-a2b9a005cc3c","Type":"ContainerDied","Data":"b583e6a531c73e87aa6415268f3a1d388dbd320722d63de823583db20218a97a"}
Sep 10 12:46:43.783998 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:43.783646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx" event={"ID":"11292fdd-e203-4393-befd-a2b9a005cc3c","Type":"ContainerStarted","Data":"dabcd669d70111c04f30747768b9a9539771f2089faa6a5a8ab4b5761341dbf4"}
Sep 10 12:46:44.927682 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:44.927656    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx"
Sep 10 12:46:45.018756 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:45.018712    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11292fdd-e203-4393-befd-a2b9a005cc3c-maas-api-service-ca\") pod \"11292fdd-e203-4393-befd-a2b9a005cc3c\" (UID: \"11292fdd-e203-4393-befd-a2b9a005cc3c\") "
Sep 10 12:46:45.019086 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:45.019062    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11292fdd-e203-4393-befd-a2b9a005cc3c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "11292fdd-e203-4393-befd-a2b9a005cc3c" (UID: "11292fdd-e203-4393-befd-a2b9a005cc3c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:46:45.120661 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:45.120534    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11292fdd-e203-4393-befd-a2b9a005cc3c-maas-api-service-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:46:45.797117 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:45.797086    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx"
Sep 10 12:46:45.797294 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:45.797124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx" event={"ID":"11292fdd-e203-4393-befd-a2b9a005cc3c","Type":"ContainerDied","Data":"dabcd669d70111c04f30747768b9a9539771f2089faa6a5a8ab4b5761341dbf4"}
Sep 10 12:46:45.797294 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:45.797179    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dabcd669d70111c04f30747768b9a9539771f2089faa6a5a8ab4b5761341dbf4"
Sep 10 12:46:47.150783 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.150746    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"]
Sep 10 12:46:47.151240 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.151210    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11292fdd-e203-4393-befd-a2b9a005cc3c" containerName="revoke-keys"
Sep 10 12:46:47.151240 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.151222    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="11292fdd-e203-4393-befd-a2b9a005cc3c" containerName="revoke-keys"
Sep 10 12:46:47.151347 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.151303    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="11292fdd-e203-4393-befd-a2b9a005cc3c" containerName="revoke-keys"
Sep 10 12:46:47.153642 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.153622    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.157658 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.157634    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1312ea\"/\"premium-w4-1312ea-kserve-self-signed-certs\""
Sep 10 12:46:47.171597 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.171569    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"]
Sep 10 12:46:47.243636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.243600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62526\" (UniqueName: \"kubernetes.io/projected/d941e342-015d-45b4-86eb-d0f2831388bf-kube-api-access-62526\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.243818 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.243652    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-home\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.243818 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.243718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-model-cache\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.243818 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.243748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d941e342-015d-45b4-86eb-d0f2831388bf-tls-certs\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.243949 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.243862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-dshm\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.243949 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.243904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-tmp-dir\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.345266 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.345230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-dshm\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.345440 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.345278    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-tmp-dir\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.345440 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.345358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62526\" (UniqueName: \"kubernetes.io/projected/d941e342-015d-45b4-86eb-d0f2831388bf-kube-api-access-62526\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.345440 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.345398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-home\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.345440 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.345433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-model-cache\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.345675 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.345459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d941e342-015d-45b4-86eb-d0f2831388bf-tls-certs\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.345807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.345780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-tmp-dir\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.345955 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.345930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-home\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.346033 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.345980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-model-cache\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.348304 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.348275    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-dshm\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.348500 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.348482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d941e342-015d-45b4-86eb-d0f2831388bf-tls-certs\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.356587 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.356525    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62526\" (UniqueName: \"kubernetes.io/projected/d941e342-015d-45b4-86eb-d0f2831388bf-kube-api-access-62526\") pod \"premium-w4-1312ea-kserve-78f9977467-789rm\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") " pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.462782 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.462746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:47.763883 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.763771    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"]
Sep 10 12:46:47.768578 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:46:47.768534    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd941e342_015d_45b4_86eb_d0f2831388bf.slice/crio-b0506b1895f3471639c9373fe4ea99c99be40709f65b7aa642edadf26190641d WatchSource:0}: Error finding container b0506b1895f3471639c9373fe4ea99c99be40709f65b7aa642edadf26190641d: Status 404 returned error can't find the container with id b0506b1895f3471639c9373fe4ea99c99be40709f65b7aa642edadf26190641d
Sep 10 12:46:47.818198 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:47.818166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm" event={"ID":"d941e342-015d-45b4-86eb-d0f2831388bf","Type":"ContainerStarted","Data":"b0506b1895f3471639c9373fe4ea99c99be40709f65b7aa642edadf26190641d"}
Sep 10 12:46:48.825940 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:48.825845    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm" event={"ID":"d941e342-015d-45b4-86eb-d0f2831388bf","Type":"ContainerStarted","Data":"e05ab99686a752d2c251e5441992996efa356f6e108618048ec21fe5d848b35e"}
Sep 10 12:46:48.872280 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:48.872223    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm" podStartSLOduration=1.872204456 podStartE2EDuration="1.872204456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:46:48.864794508 +0000 UTC m=+982.789705997" watchObservedRunningTime="2026-09-10 12:46:48.872204456 +0000 UTC m=+982.797115944"
Sep 10 12:46:50.381528 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:50.381491    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx"]
Sep 10 12:46:50.385946 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:50.385916    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3eca8974-dnbrx"]
Sep 10 12:46:50.705565 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:50.705519    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11292fdd-e203-4393-befd-a2b9a005cc3c" path="/var/lib/kubelet/pods/11292fdd-e203-4393-befd-a2b9a005cc3c/volumes"
Sep 10 12:46:52.126051 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:52.126006    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:46:57.298404 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:57.298366    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"]
Sep 10 12:46:57.298928 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:57.298603    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy" containerID="cri-o://77264db0e7f4ca15077e08b246dd2857f74a0c2b8e1985635b98573c510856d2" gracePeriod=30
Sep 10 12:46:57.463118 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:57.463080    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:57.463295 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:57.463138    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:57.475744 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:57.475718    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:46:57.898968 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:46:57.898935    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:47:00.486829 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:00.486792    2563 patch_prober.go:28] interesting pod/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 12:47:00.487204 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:00.486852    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 12:47:02.563590 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.563512    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:47:02.706240 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706204    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-workload-socket\") pod \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") "
Sep 10 12:47:02.706240 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706241    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-podinfo\") pod \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") "
Sep 10 12:47:02.706459 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706262    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istiod-ca-cert\") pod \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") "
Sep 10 12:47:02.706459 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706301    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-workload-certs\") pod \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") "
Sep 10 12:47:02.706459 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706349    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-envoy\") pod \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") "
Sep 10 12:47:02.706459 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706395    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-data\") pod \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") "
Sep 10 12:47:02.706459 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706427    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-credential-socket\") pod \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") "
Sep 10 12:47:02.706459 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706455    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfmw9\" (UniqueName: \"kubernetes.io/projected/91e9c312-8b60-41ce-8bfe-5307b08a77c3-kube-api-access-jfmw9\") pod \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") "
Sep 10 12:47:02.706794 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706494    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "91e9c312-8b60-41ce-8bfe-5307b08a77c3" (UID: "91e9c312-8b60-41ce-8bfe-5307b08a77c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:47:02.706794 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706514    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-token\") pod \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\" (UID: \"91e9c312-8b60-41ce-8bfe-5307b08a77c3\") "
Sep 10 12:47:02.706794 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706682    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "91e9c312-8b60-41ce-8bfe-5307b08a77c3" (UID: "91e9c312-8b60-41ce-8bfe-5307b08a77c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:47:02.706794 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706710    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "91e9c312-8b60-41ce-8bfe-5307b08a77c3" (UID: "91e9c312-8b60-41ce-8bfe-5307b08a77c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:47:02.707013 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706929    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-workload-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:47:02.707013 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706954    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istiod-ca-cert\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:47:02.707116 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.707074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "91e9c312-8b60-41ce-8bfe-5307b08a77c3" (UID: "91e9c312-8b60-41ce-8bfe-5307b08a77c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:47:02.707177 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.706971    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-workload-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:47:02.707425 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.707400    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-data" (OuterVolumeSpecName: "istio-data") pod "91e9c312-8b60-41ce-8bfe-5307b08a77c3" (UID: "91e9c312-8b60-41ce-8bfe-5307b08a77c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:47:02.708803 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.708774    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e9c312-8b60-41ce-8bfe-5307b08a77c3-kube-api-access-jfmw9" (OuterVolumeSpecName: "kube-api-access-jfmw9") pod "91e9c312-8b60-41ce-8bfe-5307b08a77c3" (UID: "91e9c312-8b60-41ce-8bfe-5307b08a77c3"). InnerVolumeSpecName "kube-api-access-jfmw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:47:02.708803 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.708797    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "91e9c312-8b60-41ce-8bfe-5307b08a77c3" (UID: "91e9c312-8b60-41ce-8bfe-5307b08a77c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:47:02.709234 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.709210    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "91e9c312-8b60-41ce-8bfe-5307b08a77c3" (UID: "91e9c312-8b60-41ce-8bfe-5307b08a77c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:47:02.709331 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.709233    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-token" (OuterVolumeSpecName: "istio-token") pod "91e9c312-8b60-41ce-8bfe-5307b08a77c3" (UID: "91e9c312-8b60-41ce-8bfe-5307b08a77c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:47:02.808742 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.808701    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-podinfo\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:47:02.808742 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.808733    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-envoy\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:47:02.808742 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.808747    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-data\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:47:02.808991 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.808758    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91e9c312-8b60-41ce-8bfe-5307b08a77c3-credential-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:47:02.808991 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.808771    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfmw9\" (UniqueName: \"kubernetes.io/projected/91e9c312-8b60-41ce-8bfe-5307b08a77c3-kube-api-access-jfmw9\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:47:02.808991 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.808782    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91e9c312-8b60-41ce-8bfe-5307b08a77c3-istio-token\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:47:02.915409 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.915310    2563 generic.go:356] "Generic (PLEG): container finished" podID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerID="77264db0e7f4ca15077e08b246dd2857f74a0c2b8e1985635b98573c510856d2" exitCode=0
Sep 10 12:47:02.915409 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.915375    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"
Sep 10 12:47:02.915409 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.915397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" event={"ID":"91e9c312-8b60-41ce-8bfe-5307b08a77c3","Type":"ContainerDied","Data":"77264db0e7f4ca15077e08b246dd2857f74a0c2b8e1985635b98573c510856d2"}
Sep 10 12:47:02.915716 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.915449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs" event={"ID":"91e9c312-8b60-41ce-8bfe-5307b08a77c3","Type":"ContainerDied","Data":"a91c4711d976884173616425a0601508339cc05a830878ff2034fd669b29a5fc"}
Sep 10 12:47:02.915716 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.915474    2563 scope.go:117] "RemoveContainer" containerID="77264db0e7f4ca15077e08b246dd2857f74a0c2b8e1985635b98573c510856d2"
Sep 10 12:47:02.925294 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.925279    2563 scope.go:117] "RemoveContainer" containerID="f89fce842d8ec0d4aa82c7d8ae0c54d942f006b8a822febbfde885cd25aeb97f"
Sep 10 12:47:02.933685 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.933665    2563 scope.go:117] "RemoveContainer" containerID="77264db0e7f4ca15077e08b246dd2857f74a0c2b8e1985635b98573c510856d2"
Sep 10 12:47:02.933960 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:47:02.933941    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77264db0e7f4ca15077e08b246dd2857f74a0c2b8e1985635b98573c510856d2\": container with ID starting with 77264db0e7f4ca15077e08b246dd2857f74a0c2b8e1985635b98573c510856d2 not found: ID does not exist" containerID="77264db0e7f4ca15077e08b246dd2857f74a0c2b8e1985635b98573c510856d2"
Sep 10 12:47:02.934008 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.933969    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77264db0e7f4ca15077e08b246dd2857f74a0c2b8e1985635b98573c510856d2"} err="failed to get container status \"77264db0e7f4ca15077e08b246dd2857f74a0c2b8e1985635b98573c510856d2\": rpc error: code = NotFound desc = could not find container \"77264db0e7f4ca15077e08b246dd2857f74a0c2b8e1985635b98573c510856d2\": container with ID starting with 77264db0e7f4ca15077e08b246dd2857f74a0c2b8e1985635b98573c510856d2 not found: ID does not exist"
Sep 10 12:47:02.934008 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.933987    2563 scope.go:117] "RemoveContainer" containerID="f89fce842d8ec0d4aa82c7d8ae0c54d942f006b8a822febbfde885cd25aeb97f"
Sep 10 12:47:02.934207 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:47:02.934191    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89fce842d8ec0d4aa82c7d8ae0c54d942f006b8a822febbfde885cd25aeb97f\": container with ID starting with f89fce842d8ec0d4aa82c7d8ae0c54d942f006b8a822febbfde885cd25aeb97f not found: ID does not exist" containerID="f89fce842d8ec0d4aa82c7d8ae0c54d942f006b8a822febbfde885cd25aeb97f"
Sep 10 12:47:02.934252 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.934210    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89fce842d8ec0d4aa82c7d8ae0c54d942f006b8a822febbfde885cd25aeb97f"} err="failed to get container status \"f89fce842d8ec0d4aa82c7d8ae0c54d942f006b8a822febbfde885cd25aeb97f\": rpc error: code = NotFound desc = could not find container \"f89fce842d8ec0d4aa82c7d8ae0c54d942f006b8a822febbfde885cd25aeb97f\": container with ID starting with f89fce842d8ec0d4aa82c7d8ae0c54d942f006b8a822febbfde885cd25aeb97f not found: ID does not exist"
Sep 10 12:47:02.949941 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.949914    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"]
Sep 10 12:47:02.953272 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:02.953251    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-5abff0-openshift-default-9c9c9b44b-26pqs"]
Sep 10 12:47:04.564216 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.564180    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"]
Sep 10 12:47:04.564837 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.564816    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy"
Sep 10 12:47:04.564939 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.564840    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy"
Sep 10 12:47:04.564939 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.564857    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy"
Sep 10 12:47:04.564939 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.564873    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy"
Sep 10 12:47:04.565090 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.565011    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy"
Sep 10 12:47:04.565090 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.565028    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" containerName="istio-proxy"
Sep 10 12:47:04.569861 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.569839    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"
Sep 10 12:47:04.574693 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.574670    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-2d4635\""
Sep 10 12:47:04.578289 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.578271    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-2d4635-dockercfg-8snnw\""
Sep 10 12:47:04.597596 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.597545    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"]
Sep 10 12:47:04.706919 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.706886    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e9c312-8b60-41ce-8bfe-5307b08a77c3" path="/var/lib/kubelet/pods/91e9c312-8b60-41ce-8bfe-5307b08a77c3/volumes"
Sep 10 12:47:04.729175 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.729138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vg5q\" (UniqueName: \"kubernetes.io/projected/ba6e7455-925f-4c81-83dd-7c7170bed9a3-kube-api-access-5vg5q\") pod \"payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws\" (UID: \"ba6e7455-925f-4c81-83dd-7c7170bed9a3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"
Sep 10 12:47:04.729342 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.729265    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6e7455-925f-4c81-83dd-7c7170bed9a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws\" (UID: \"ba6e7455-925f-4c81-83dd-7c7170bed9a3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"
Sep 10 12:47:04.829965 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.829871    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6e7455-925f-4c81-83dd-7c7170bed9a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws\" (UID: \"ba6e7455-925f-4c81-83dd-7c7170bed9a3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"
Sep 10 12:47:04.830245 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.830217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vg5q\" (UniqueName: \"kubernetes.io/projected/ba6e7455-925f-4c81-83dd-7c7170bed9a3-kube-api-access-5vg5q\") pod \"payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws\" (UID: \"ba6e7455-925f-4c81-83dd-7c7170bed9a3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"
Sep 10 12:47:04.830665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.830641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6e7455-925f-4c81-83dd-7c7170bed9a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws\" (UID: \"ba6e7455-925f-4c81-83dd-7c7170bed9a3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"
Sep 10 12:47:04.852517 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.852485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vg5q\" (UniqueName: \"kubernetes.io/projected/ba6e7455-925f-4c81-83dd-7c7170bed9a3-kube-api-access-5vg5q\") pod \"payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws\" (UID: \"ba6e7455-925f-4c81-83dd-7c7170bed9a3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"
Sep 10 12:47:04.880743 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:04.880703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"
Sep 10 12:47:05.177260 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:05.177227    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"]
Sep 10 12:47:05.194465 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:47:05.194433    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6e7455_925f_4c81_83dd_7c7170bed9a3.slice/crio-97d1589e9f0ece58ba763eeaeeee3499faf480a7362fb0eedcd91641ddf3987c WatchSource:0}: Error finding container 97d1589e9f0ece58ba763eeaeeee3499faf480a7362fb0eedcd91641ddf3987c: Status 404 returned error can't find the container with id 97d1589e9f0ece58ba763eeaeeee3499faf480a7362fb0eedcd91641ddf3987c
Sep 10 12:47:05.935911 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:05.935862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws" event={"ID":"ba6e7455-925f-4c81-83dd-7c7170bed9a3","Type":"ContainerStarted","Data":"c88d3d012c3519287b69680e8aa52e845e71d4188b7832f8523ca241b6836e1c"}
Sep 10 12:47:05.936299 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:05.935930    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws" event={"ID":"ba6e7455-925f-4c81-83dd-7c7170bed9a3","Type":"ContainerStarted","Data":"97d1589e9f0ece58ba763eeaeeee3499faf480a7362fb0eedcd91641ddf3987c"}
Sep 10 12:47:05.936299 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:05.935950    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"
Sep 10 12:47:05.964971 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:05.964916    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws" podStartSLOduration=1.964900318 podStartE2EDuration="1.964900318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:47:05.963181713 +0000 UTC m=+999.888093203" watchObservedRunningTime="2026-09-10 12:47:05.964900318 +0000 UTC m=+999.889811827"
Sep 10 12:47:16.943319 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:16.943284    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"
Sep 10 12:47:17.745460 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:17.745426    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:47:27.979403 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:27.979356    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"]
Sep 10 12:47:27.984477 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:27.984441    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:27.995224 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:27.995201    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-30919a\"/\"premium-w0-30919a-kserve-self-signed-certs\""
Sep 10 12:47:28.057005 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.056971    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"]
Sep 10 12:47:28.067175 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.067138    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-home\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.067339 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.067250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-model-cache\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.067389 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.067347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a4386e-53a8-419a-be7e-9bb491d947b9-tls-certs\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.067437 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.067387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-tmp-dir\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.067475 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.067461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-dshm\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.067518 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.067502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmmkf\" (UniqueName: \"kubernetes.io/projected/72a4386e-53a8-419a-be7e-9bb491d947b9-kube-api-access-cmmkf\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.168799 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.168764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a4386e-53a8-419a-be7e-9bb491d947b9-tls-certs\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.169006 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.168814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-tmp-dir\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.169006 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.168861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-dshm\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.169006 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.168910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmmkf\" (UniqueName: \"kubernetes.io/projected/72a4386e-53a8-419a-be7e-9bb491d947b9-kube-api-access-cmmkf\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.169006 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.168960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-home\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.169240 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.169025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-model-cache\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.169673 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.169644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-tmp-dir\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.169800 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.169700    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-model-cache\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.169800 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.169713    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-home\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.172121 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.172097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-dshm\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.172248 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.172139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a4386e-53a8-419a-be7e-9bb491d947b9-tls-certs\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.187636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.187607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmmkf\" (UniqueName: \"kubernetes.io/projected/72a4386e-53a8-419a-be7e-9bb491d947b9-kube-api-access-cmmkf\") pod \"premium-w0-30919a-kserve-67f56cc8fc-kwl8s\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") " pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.295817 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.295730    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:28.605493 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:28.605463    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"]
Sep 10 12:47:28.608105 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:47:28.608080    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72a4386e_53a8_419a_be7e_9bb491d947b9.slice/crio-f932f76571ea72af970c37ba57f53ce6777caa4d7e2b4b780b8652b2f5c76cd1 WatchSource:0}: Error finding container f932f76571ea72af970c37ba57f53ce6777caa4d7e2b4b780b8652b2f5c76cd1: Status 404 returned error can't find the container with id f932f76571ea72af970c37ba57f53ce6777caa4d7e2b4b780b8652b2f5c76cd1
Sep 10 12:47:29.086674 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:29.086630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s" event={"ID":"72a4386e-53a8-419a-be7e-9bb491d947b9","Type":"ContainerStarted","Data":"a50aa57a8b8684a530b17aeb03e284d09e173176254236fffe92306a88561abd"}
Sep 10 12:47:29.087088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:29.086688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s" event={"ID":"72a4386e-53a8-419a-be7e-9bb491d947b9","Type":"ContainerStarted","Data":"f932f76571ea72af970c37ba57f53ce6777caa4d7e2b4b780b8652b2f5c76cd1"}
Sep 10 12:47:29.137067 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:29.137010    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s" podStartSLOduration=2.136994435 podStartE2EDuration="2.136994435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:47:29.134256511 +0000 UTC m=+1023.059167999" watchObservedRunningTime="2026-09-10 12:47:29.136994435 +0000 UTC m=+1023.061905956"
Sep 10 12:47:34.329775 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:34.329736    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:47:38.296829 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:38.296792    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:38.297233 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:38.296849    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:38.308826 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:38.308794    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:39.167538 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:39.167504    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:47:50.182488 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.182455    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"]
Sep 10 12:47:50.187068 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.187042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.190702 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.190673    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-2d4635\"/\"default-dockercfg-lnb5n\""
Sep 10 12:47:50.190832 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.190741    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-2d4635\"/\"openshift-service-ca.crt\""
Sep 10 12:47:50.190832 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.190776    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-2d4635\"/\"explicit-model-2d4635-kserve-self-signed-certs\""
Sep 10 12:47:50.190957 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.190901    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-2d4635\"/\"kube-root-ca.crt\""
Sep 10 12:47:50.236668 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.236635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"]
Sep 10 12:47:50.309869 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.309834    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-home\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.310049 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.309891    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-tmp-dir\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.310049 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.309964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-dshm\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.310049 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.309998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-model-cache\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.310049 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.310044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902ee064-ff13-4dce-944d-7dad59f329b4-tls-certs\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.310226 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.310064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfxp\" (UniqueName: \"kubernetes.io/projected/902ee064-ff13-4dce-944d-7dad59f329b4-kube-api-access-6xfxp\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.410931 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.410895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-home\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.411102 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.410942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-tmp-dir\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.411102 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.410975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-dshm\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.411102 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.410990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-model-cache\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.411102 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.411028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902ee064-ff13-4dce-944d-7dad59f329b4-tls-certs\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.411102 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.411046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfxp\" (UniqueName: \"kubernetes.io/projected/902ee064-ff13-4dce-944d-7dad59f329b4-kube-api-access-6xfxp\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.411374 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.411343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-home\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.411374 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.411358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-tmp-dir\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.411482 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.411406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-model-cache\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.413420 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.413395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-dshm\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.413537 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.413524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902ee064-ff13-4dce-944d-7dad59f329b4-tls-certs\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.430881 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.430862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfxp\" (UniqueName: \"kubernetes.io/projected/902ee064-ff13-4dce-944d-7dad59f329b4-kube-api-access-6xfxp\") pod \"explicit-model-2d4635-kserve-69984bf49c-gwj7c\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") " pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.499273 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.499174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:47:50.801773 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:50.801747    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"]
Sep 10 12:47:50.804020 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:47:50.803994    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod902ee064_ff13_4dce_944d_7dad59f329b4.slice/crio-0a94d92e9dbc81d91dd749e728eaaf26b49d6274ec6098ac3f3e5c9b88052ad9 WatchSource:0}: Error finding container 0a94d92e9dbc81d91dd749e728eaaf26b49d6274ec6098ac3f3e5c9b88052ad9: Status 404 returned error can't find the container with id 0a94d92e9dbc81d91dd749e728eaaf26b49d6274ec6098ac3f3e5c9b88052ad9
Sep 10 12:47:51.229907 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:51.229867    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c" event={"ID":"902ee064-ff13-4dce-944d-7dad59f329b4","Type":"ContainerStarted","Data":"ed388ff8aa9787183115103731dd6004b8233b3ab94747d585f2d43c91b6bff6"}
Sep 10 12:47:51.229907 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:51.229915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c" event={"ID":"902ee064-ff13-4dce-944d-7dad59f329b4","Type":"ContainerStarted","Data":"0a94d92e9dbc81d91dd749e728eaaf26b49d6274ec6098ac3f3e5c9b88052ad9"}
Sep 10 12:47:51.262123 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:51.262067    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c" podStartSLOduration=1.2620513070000001 podStartE2EDuration="1.262051307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:47:51.261201476 +0000 UTC m=+1045.186112964" watchObservedRunningTime="2026-09-10 12:47:51.262051307 +0000 UTC m=+1045.186962795"
Sep 10 12:47:52.182008 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:47:52.181974    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:48:00.499994 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:00.499950    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:48:00.499994 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:00.500005    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:48:00.512260 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:00.512231    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:48:01.297889 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:01.297860    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:48:06.826704 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:06.826671    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"]
Sep 10 12:48:06.830657 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:06.830635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:06.833204 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:06.833182    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1312ea\"/\"distinct-w4-1312ea-kserve-self-signed-certs\""
Sep 10 12:48:06.840930 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:06.840908    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"]
Sep 10 12:48:06.985960 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:06.985924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-home\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:06.985960 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:06.985958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f6d5312-fdb6-40e6-80bb-703f052a0910-tls-certs\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:06.986173 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:06.985981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-dshm\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:06.986173 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:06.986068    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-model-cache\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:06.986173 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:06.986150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcpmq\" (UniqueName: \"kubernetes.io/projected/6f6d5312-fdb6-40e6-80bb-703f052a0910-kube-api-access-lcpmq\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:06.986294 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:06.986201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-tmp-dir\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.087486 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.087386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-tmp-dir\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.087683 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.087494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-home\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.087683 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.087520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f6d5312-fdb6-40e6-80bb-703f052a0910-tls-certs\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.087683 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.087545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-dshm\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.087860 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.087684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-model-cache\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.087860 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.087800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-tmp-dir\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.087860 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.087806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcpmq\" (UniqueName: \"kubernetes.io/projected/6f6d5312-fdb6-40e6-80bb-703f052a0910-kube-api-access-lcpmq\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.087860 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.087851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-home\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.088024 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.087999    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-model-cache\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.089799 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.089780    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-dshm\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.090195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.090162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f6d5312-fdb6-40e6-80bb-703f052a0910-tls-certs\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.098434 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.098406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcpmq\" (UniqueName: \"kubernetes.io/projected/6f6d5312-fdb6-40e6-80bb-703f052a0910-kube-api-access-lcpmq\") pod \"distinct-w4-1312ea-kserve-764d8f7746-69qfc\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.140588 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.140560    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:07.432709 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:07.432676    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"]
Sep 10 12:48:07.433925 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:48:07.433898    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f6d5312_fdb6_40e6_80bb_703f052a0910.slice/crio-76f51cb648d3067afbc14045aa7b01de5e9731c795df49b4a3ee4f91c8655e76 WatchSource:0}: Error finding container 76f51cb648d3067afbc14045aa7b01de5e9731c795df49b4a3ee4f91c8655e76: Status 404 returned error can't find the container with id 76f51cb648d3067afbc14045aa7b01de5e9731c795df49b4a3ee4f91c8655e76
Sep 10 12:48:08.329379 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:08.329336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc" event={"ID":"6f6d5312-fdb6-40e6-80bb-703f052a0910","Type":"ContainerStarted","Data":"1af326bc72279cb50d9312f5fa472ec6baaa299e92ca6eec595b5e0d2c1bf41c"}
Sep 10 12:48:08.329798 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:08.329391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc" event={"ID":"6f6d5312-fdb6-40e6-80bb-703f052a0910","Type":"ContainerStarted","Data":"76f51cb648d3067afbc14045aa7b01de5e9731c795df49b4a3ee4f91c8655e76"}
Sep 10 12:48:08.365612 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:08.365545    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc" podStartSLOduration=2.365529767 podStartE2EDuration="2.365529767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:48:08.363880813 +0000 UTC m=+1062.288792299" watchObservedRunningTime="2026-09-10 12:48:08.365529767 +0000 UTC m=+1062.290441254"
Sep 10 12:48:14.475402 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:14.475364    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:48:17.140714 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:17.140675    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:17.140714 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:17.140724    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:17.153431 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:17.153404    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:17.402332 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:17.402245    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:48:24.268865 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:24.268781    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2"]
Sep 10 12:48:24.271691 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:24.271674    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2"
Sep 10 12:48:24.290282 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:24.290257    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:48:24.290738 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:24.290722    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:48:24.334966 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:24.334938    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2"]
Sep 10 12:48:24.357646 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:24.357616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5fc8d8-e12e-4424-b3b8-25149c97fc95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2\" (UID: \"2c5fc8d8-e12e-4424-b3b8-25149c97fc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2"
Sep 10 12:48:24.458333 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:24.458303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5fc8d8-e12e-4424-b3b8-25149c97fc95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2\" (UID: \"2c5fc8d8-e12e-4424-b3b8-25149c97fc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2"
Sep 10 12:48:24.458990 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:24.458968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5fc8d8-e12e-4424-b3b8-25149c97fc95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2\" (UID: \"2c5fc8d8-e12e-4424-b3b8-25149c97fc95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2"
Sep 10 12:48:24.582594 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:24.582484    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2"
Sep 10 12:48:24.858341 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:24.858315    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2"]
Sep 10 12:48:24.860488 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:48:24.860462    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5fc8d8_e12e_4424_b3b8_25149c97fc95.slice/crio-391cb54859b9cc53a7afd0d45edfeba2f8813caff3b3e9bc7c6bccd2594400d8 WatchSource:0}: Error finding container 391cb54859b9cc53a7afd0d45edfeba2f8813caff3b3e9bc7c6bccd2594400d8: Status 404 returned error can't find the container with id 391cb54859b9cc53a7afd0d45edfeba2f8813caff3b3e9bc7c6bccd2594400d8
Sep 10 12:48:25.432400 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:25.432362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2" event={"ID":"2c5fc8d8-e12e-4424-b3b8-25149c97fc95","Type":"ContainerStarted","Data":"fbc1c7964774422d54ad8cc5a04a10a684df559901aa1ba05ddc13f2f07f7385"}
Sep 10 12:48:25.432773 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:25.432420    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2" event={"ID":"2c5fc8d8-e12e-4424-b3b8-25149c97fc95","Type":"ContainerStarted","Data":"391cb54859b9cc53a7afd0d45edfeba2f8813caff3b3e9bc7c6bccd2594400d8"}
Sep 10 12:48:25.453514 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:25.453448    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2" podStartSLOduration=1.453428496 podStartE2EDuration="1.453428496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:48:25.449471469 +0000 UTC m=+1079.374382958" watchObservedRunningTime="2026-09-10 12:48:25.453428496 +0000 UTC m=+1079.378339986"
Sep 10 12:48:26.439401 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:26.439366    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c5fc8d8-e12e-4424-b3b8-25149c97fc95" containerID="fbc1c7964774422d54ad8cc5a04a10a684df559901aa1ba05ddc13f2f07f7385" exitCode=0
Sep 10 12:48:26.439801 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:26.439453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2" event={"ID":"2c5fc8d8-e12e-4424-b3b8-25149c97fc95","Type":"ContainerDied","Data":"fbc1c7964774422d54ad8cc5a04a10a684df559901aa1ba05ddc13f2f07f7385"}
Sep 10 12:48:27.591650 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:27.591624    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2"
Sep 10 12:48:27.689391 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:27.689341    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5fc8d8-e12e-4424-b3b8-25149c97fc95-maas-api-service-ca\") pod \"2c5fc8d8-e12e-4424-b3b8-25149c97fc95\" (UID: \"2c5fc8d8-e12e-4424-b3b8-25149c97fc95\") "
Sep 10 12:48:27.689723 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:27.689698    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5fc8d8-e12e-4424-b3b8-25149c97fc95-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c5fc8d8-e12e-4424-b3b8-25149c97fc95" (UID: "2c5fc8d8-e12e-4424-b3b8-25149c97fc95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:48:27.790231 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:27.790133    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c5fc8d8-e12e-4424-b3b8-25149c97fc95-maas-api-service-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:48:28.454109 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:28.454076    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2"
Sep 10 12:48:28.454272 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:28.454104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2" event={"ID":"2c5fc8d8-e12e-4424-b3b8-25149c97fc95","Type":"ContainerDied","Data":"391cb54859b9cc53a7afd0d45edfeba2f8813caff3b3e9bc7c6bccd2594400d8"}
Sep 10 12:48:28.454272 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:28.454192    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="391cb54859b9cc53a7afd0d45edfeba2f8813caff3b3e9bc7c6bccd2594400d8"
Sep 10 12:48:30.263303 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:30.263266    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"]
Sep 10 12:48:34.946893 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:34.946857    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"]
Sep 10 12:48:34.947445 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:34.947152    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws" podUID="ba6e7455-925f-4c81-83dd-7c7170bed9a3" containerName="payload-pre-processing" containerID="cri-o://c88d3d012c3519287b69680e8aa52e845e71d4188b7832f8523ca241b6836e1c" gracePeriod=30
Sep 10 12:48:35.211416 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.211349    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"
Sep 10 12:48:35.259288 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.259252    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2"]
Sep 10 12:48:35.264144 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.264115    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-2d4635-xn8b2"]
Sep 10 12:48:35.270927 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.270892    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6e7455-925f-4c81-83dd-7c7170bed9a3-plugins-config-volume\") pod \"ba6e7455-925f-4c81-83dd-7c7170bed9a3\" (UID: \"ba6e7455-925f-4c81-83dd-7c7170bed9a3\") "
Sep 10 12:48:35.271039 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.270995    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vg5q\" (UniqueName: \"kubernetes.io/projected/ba6e7455-925f-4c81-83dd-7c7170bed9a3-kube-api-access-5vg5q\") pod \"ba6e7455-925f-4c81-83dd-7c7170bed9a3\" (UID: \"ba6e7455-925f-4c81-83dd-7c7170bed9a3\") "
Sep 10 12:48:35.271250 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.271228    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6e7455-925f-4c81-83dd-7c7170bed9a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ba6e7455-925f-4c81-83dd-7c7170bed9a3" (UID: "ba6e7455-925f-4c81-83dd-7c7170bed9a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:48:35.273049 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.273028    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6e7455-925f-4c81-83dd-7c7170bed9a3-kube-api-access-5vg5q" (OuterVolumeSpecName: "kube-api-access-5vg5q") pod "ba6e7455-925f-4c81-83dd-7c7170bed9a3" (UID: "ba6e7455-925f-4c81-83dd-7c7170bed9a3"). InnerVolumeSpecName "kube-api-access-5vg5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:48:35.372366 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.372328    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vg5q\" (UniqueName: \"kubernetes.io/projected/ba6e7455-925f-4c81-83dd-7c7170bed9a3-kube-api-access-5vg5q\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:48:35.372366 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.372360    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6e7455-925f-4c81-83dd-7c7170bed9a3-plugins-config-volume\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:48:35.498756 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.498663    2563 generic.go:356] "Generic (PLEG): container finished" podID="ba6e7455-925f-4c81-83dd-7c7170bed9a3" containerID="c88d3d012c3519287b69680e8aa52e845e71d4188b7832f8523ca241b6836e1c" exitCode=0
Sep 10 12:48:35.498756 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.498709    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"
Sep 10 12:48:35.498756 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.498742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws" event={"ID":"ba6e7455-925f-4c81-83dd-7c7170bed9a3","Type":"ContainerDied","Data":"c88d3d012c3519287b69680e8aa52e845e71d4188b7832f8523ca241b6836e1c"}
Sep 10 12:48:35.499019 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.498795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws" event={"ID":"ba6e7455-925f-4c81-83dd-7c7170bed9a3","Type":"ContainerDied","Data":"97d1589e9f0ece58ba763eeaeeee3499faf480a7362fb0eedcd91641ddf3987c"}
Sep 10 12:48:35.499019 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.498818    2563 scope.go:117] "RemoveContainer" containerID="c88d3d012c3519287b69680e8aa52e845e71d4188b7832f8523ca241b6836e1c"
Sep 10 12:48:35.508247 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.508230    2563 scope.go:117] "RemoveContainer" containerID="c88d3d012c3519287b69680e8aa52e845e71d4188b7832f8523ca241b6836e1c"
Sep 10 12:48:35.508507 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:48:35.508490    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88d3d012c3519287b69680e8aa52e845e71d4188b7832f8523ca241b6836e1c\": container with ID starting with c88d3d012c3519287b69680e8aa52e845e71d4188b7832f8523ca241b6836e1c not found: ID does not exist" containerID="c88d3d012c3519287b69680e8aa52e845e71d4188b7832f8523ca241b6836e1c"
Sep 10 12:48:35.508571 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.508515    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88d3d012c3519287b69680e8aa52e845e71d4188b7832f8523ca241b6836e1c"} err="failed to get container status \"c88d3d012c3519287b69680e8aa52e845e71d4188b7832f8523ca241b6836e1c\": rpc error: code = NotFound desc = could not find container \"c88d3d012c3519287b69680e8aa52e845e71d4188b7832f8523ca241b6836e1c\": container with ID starting with c88d3d012c3519287b69680e8aa52e845e71d4188b7832f8523ca241b6836e1c not found: ID does not exist"
Sep 10 12:48:35.531436 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.531407    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"]
Sep 10 12:48:35.543084 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:35.543055    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2d4635-7c5869bdfd-tlkws"]
Sep 10 12:48:36.705649 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:36.705597    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c5fc8d8-e12e-4424-b3b8-25149c97fc95" path="/var/lib/kubelet/pods/2c5fc8d8-e12e-4424-b3b8-25149c97fc95/volumes"
Sep 10 12:48:36.706037 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:36.706023    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba6e7455-925f-4c81-83dd-7c7170bed9a3" path="/var/lib/kubelet/pods/ba6e7455-925f-4c81-83dd-7c7170bed9a3/volumes"
Sep 10 12:48:45.283014 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.282963    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c" podUID="902ee064-ff13-4dce-944d-7dad59f329b4" containerName="main" containerID="cri-o://ed388ff8aa9787183115103731dd6004b8233b3ab94747d585f2d43c91b6bff6" gracePeriod=45
Sep 10 12:48:45.532854 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.532828    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:48:45.566121 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.566025    2563 generic.go:356] "Generic (PLEG): container finished" podID="902ee064-ff13-4dce-944d-7dad59f329b4" containerID="ed388ff8aa9787183115103731dd6004b8233b3ab94747d585f2d43c91b6bff6" exitCode=0
Sep 10 12:48:45.566121 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.566067    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"
Sep 10 12:48:45.566121 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.566078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c" event={"ID":"902ee064-ff13-4dce-944d-7dad59f329b4","Type":"ContainerDied","Data":"ed388ff8aa9787183115103731dd6004b8233b3ab94747d585f2d43c91b6bff6"}
Sep 10 12:48:45.566121 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.566120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c" event={"ID":"902ee064-ff13-4dce-944d-7dad59f329b4","Type":"ContainerDied","Data":"0a94d92e9dbc81d91dd749e728eaaf26b49d6274ec6098ac3f3e5c9b88052ad9"}
Sep 10 12:48:45.566497 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.566145    2563 scope.go:117] "RemoveContainer" containerID="ed388ff8aa9787183115103731dd6004b8233b3ab94747d585f2d43c91b6bff6"
Sep 10 12:48:45.575527 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.575499    2563 scope.go:117] "RemoveContainer" containerID="ed388ff8aa9787183115103731dd6004b8233b3ab94747d585f2d43c91b6bff6"
Sep 10 12:48:45.575903 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:48:45.575879    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed388ff8aa9787183115103731dd6004b8233b3ab94747d585f2d43c91b6bff6\": container with ID starting with ed388ff8aa9787183115103731dd6004b8233b3ab94747d585f2d43c91b6bff6 not found: ID does not exist" containerID="ed388ff8aa9787183115103731dd6004b8233b3ab94747d585f2d43c91b6bff6"
Sep 10 12:48:45.575999 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.575916    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed388ff8aa9787183115103731dd6004b8233b3ab94747d585f2d43c91b6bff6"} err="failed to get container status \"ed388ff8aa9787183115103731dd6004b8233b3ab94747d585f2d43c91b6bff6\": rpc error: code = NotFound desc = could not find container \"ed388ff8aa9787183115103731dd6004b8233b3ab94747d585f2d43c91b6bff6\": container with ID starting with ed388ff8aa9787183115103731dd6004b8233b3ab94747d585f2d43c91b6bff6 not found: ID does not exist"
Sep 10 12:48:45.680633 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.680602    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-model-cache\") pod \"902ee064-ff13-4dce-944d-7dad59f329b4\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") "
Sep 10 12:48:45.680777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.680669    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xfxp\" (UniqueName: \"kubernetes.io/projected/902ee064-ff13-4dce-944d-7dad59f329b4-kube-api-access-6xfxp\") pod \"902ee064-ff13-4dce-944d-7dad59f329b4\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") "
Sep 10 12:48:45.680777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.680710    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-dshm\") pod \"902ee064-ff13-4dce-944d-7dad59f329b4\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") "
Sep 10 12:48:45.680777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.680755    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-home\") pod \"902ee064-ff13-4dce-944d-7dad59f329b4\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") "
Sep 10 12:48:45.680962 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.680783    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-tmp-dir\") pod \"902ee064-ff13-4dce-944d-7dad59f329b4\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") "
Sep 10 12:48:45.680962 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.680816    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902ee064-ff13-4dce-944d-7dad59f329b4-tls-certs\") pod \"902ee064-ff13-4dce-944d-7dad59f329b4\" (UID: \"902ee064-ff13-4dce-944d-7dad59f329b4\") "
Sep 10 12:48:45.681095 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.681071    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-home" (OuterVolumeSpecName: "home") pod "902ee064-ff13-4dce-944d-7dad59f329b4" (UID: "902ee064-ff13-4dce-944d-7dad59f329b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:48:45.681154 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.681099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "902ee064-ff13-4dce-944d-7dad59f329b4" (UID: "902ee064-ff13-4dce-944d-7dad59f329b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:48:45.681256 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.680957    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "902ee064-ff13-4dce-944d-7dad59f329b4" (UID: "902ee064-ff13-4dce-944d-7dad59f329b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:48:45.681337 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.681321    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:48:45.681377 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.681344    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:48:45.683093 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.683062    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-dshm" (OuterVolumeSpecName: "dshm") pod "902ee064-ff13-4dce-944d-7dad59f329b4" (UID: "902ee064-ff13-4dce-944d-7dad59f329b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:48:45.683093 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.683075    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902ee064-ff13-4dce-944d-7dad59f329b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "902ee064-ff13-4dce-944d-7dad59f329b4" (UID: "902ee064-ff13-4dce-944d-7dad59f329b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:48:45.683238 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.683106    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902ee064-ff13-4dce-944d-7dad59f329b4-kube-api-access-6xfxp" (OuterVolumeSpecName: "kube-api-access-6xfxp") pod "902ee064-ff13-4dce-944d-7dad59f329b4" (UID: "902ee064-ff13-4dce-944d-7dad59f329b4"). InnerVolumeSpecName "kube-api-access-6xfxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:48:45.781989 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.781949    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:48:45.781989 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.781982    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xfxp\" (UniqueName: \"kubernetes.io/projected/902ee064-ff13-4dce-944d-7dad59f329b4-kube-api-access-6xfxp\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:48:45.781989 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.781992    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/902ee064-ff13-4dce-944d-7dad59f329b4-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:48:45.782244 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.782002    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/902ee064-ff13-4dce-944d-7dad59f329b4-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:48:45.901261 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.901225    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"]
Sep 10 12:48:45.915481 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:45.915447    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-2d4635/explicit-model-2d4635-kserve-69984bf49c-gwj7c"]
Sep 10 12:48:46.706505 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:46.706468    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="902ee064-ff13-4dce-944d-7dad59f329b4" path="/var/lib/kubelet/pods/902ee064-ff13-4dce-944d-7dad59f329b4/volumes"
Sep 10 12:48:49.403608 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.403575    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"]
Sep 10 12:48:49.404021 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.404007    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="902ee064-ff13-4dce-944d-7dad59f329b4" containerName="main"
Sep 10 12:48:49.404066 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.404023    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="902ee064-ff13-4dce-944d-7dad59f329b4" containerName="main"
Sep 10 12:48:49.404066 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.404049    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba6e7455-925f-4c81-83dd-7c7170bed9a3" containerName="payload-pre-processing"
Sep 10 12:48:49.404066 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.404054    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba6e7455-925f-4c81-83dd-7c7170bed9a3" containerName="payload-pre-processing"
Sep 10 12:48:49.404163 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.404067    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c5fc8d8-e12e-4424-b3b8-25149c97fc95" containerName="revoke-keys"
Sep 10 12:48:49.404163 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.404073    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c5fc8d8-e12e-4424-b3b8-25149c97fc95" containerName="revoke-keys"
Sep 10 12:48:49.404163 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.404149    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c5fc8d8-e12e-4424-b3b8-25149c97fc95" containerName="revoke-keys"
Sep 10 12:48:49.404163 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.404161    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba6e7455-925f-4c81-83dd-7c7170bed9a3" containerName="payload-pre-processing"
Sep 10 12:48:49.404280 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.404170    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="902ee064-ff13-4dce-944d-7dad59f329b4" containerName="main"
Sep 10 12:48:49.408671 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.408650    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.411916 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.411892    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-340098-openshift-default-dockercfg-smxpr\""
Sep 10 12:48:49.423395 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.423370    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"]
Sep 10 12:48:49.519814 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.519779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6kz\" (UniqueName: \"kubernetes.io/projected/a707f87e-7b63-45d9-a092-adc6f38fbfca-kube-api-access-ck6kz\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.519981 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.519825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-podinfo\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.519981 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.519915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-credential-socket\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.519981 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.519960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-workload-certs\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.520156 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.519989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-workload-socket\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.520156 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.520006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a707f87e-7b63-45d9-a092-adc6f38fbfca-istiod-ca-cert\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.520156 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.520032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-data\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.520156 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.520098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-token\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.520330 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.520160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-envoy\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.621226 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.621189    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-workload-socket\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.621226 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.621230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a707f87e-7b63-45d9-a092-adc6f38fbfca-istiod-ca-cert\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.621448 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.621265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-data\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.621448 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.621328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-token\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.621448 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.621356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-envoy\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.621448 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.621404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6kz\" (UniqueName: \"kubernetes.io/projected/a707f87e-7b63-45d9-a092-adc6f38fbfca-kube-api-access-ck6kz\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.621448 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.621439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-podinfo\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.621770 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.621714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-credential-socket\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.622059 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.622035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a707f87e-7b63-45d9-a092-adc6f38fbfca-istiod-ca-cert\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.622135 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.621792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-data\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.622135 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.621960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-credential-socket\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.622135 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.621731    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-workload-socket\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.622135 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.622048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-workload-certs\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.622352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.622279    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-workload-certs\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.623674 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.623652    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-envoy\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.623929 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.623910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-podinfo\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.635196 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.635169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6kz\" (UniqueName: \"kubernetes.io/projected/a707f87e-7b63-45d9-a092-adc6f38fbfca-kube-api-access-ck6kz\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.640310 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.640285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-token\") pod \"e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") " pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:49.723110 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:49.723075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:50.037229 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:50.037199    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"]
Sep 10 12:48:50.038381 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:48:50.038349    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda707f87e_7b63_45d9_a092_adc6f38fbfca.slice/crio-707d0d9bc3329dddd528b20d5b0f8f58a1e2d829736a9f4afed45b539052790d WatchSource:0}: Error finding container 707d0d9bc3329dddd528b20d5b0f8f58a1e2d829736a9f4afed45b539052790d: Status 404 returned error can't find the container with id 707d0d9bc3329dddd528b20d5b0f8f58a1e2d829736a9f4afed45b539052790d
Sep 10 12:48:50.040699 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:50.040663    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:48:50.040768 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:50.040733    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:48:50.040768 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:50.040761    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:48:50.598876 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:50.598836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs" event={"ID":"a707f87e-7b63-45d9-a092-adc6f38fbfca","Type":"ContainerStarted","Data":"95cfe66b9ba0513ad876854cf710edc7d74fa7ae09e7241d644dcdfb0427873f"}
Sep 10 12:48:50.599273 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:50.598905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs" event={"ID":"a707f87e-7b63-45d9-a092-adc6f38fbfca","Type":"ContainerStarted","Data":"707d0d9bc3329dddd528b20d5b0f8f58a1e2d829736a9f4afed45b539052790d"}
Sep 10 12:48:50.647982 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:50.647919    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs" podStartSLOduration=1.647887237 podStartE2EDuration="1.647887237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:48:50.644404664 +0000 UTC m=+1104.569316149" watchObservedRunningTime="2026-09-10 12:48:50.647887237 +0000 UTC m=+1104.572798726"
Sep 10 12:48:50.724037 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:50.724004    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:51.722972 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:51.722940    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:48:51.728156 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:51.728129    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:52.612659 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:52.612625    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:48:52.613726 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:48:52.613706    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:49:22.861077 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:22.861044    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f"]
Sep 10 12:49:22.864884 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:22.864862    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f"
Sep 10 12:49:22.870022 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:22.869994    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:49:22.870162 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:22.870067    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:49:22.876966 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:22.876922    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f"]
Sep 10 12:49:22.985010 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:22.984975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7ec649-b8e8-4906-a1b3-4e7ddefca857-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-006b1977-czd4f\" (UID: \"ba7ec649-b8e8-4906-a1b3-4e7ddefca857\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f"
Sep 10 12:49:23.086638 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:23.086592    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7ec649-b8e8-4906-a1b3-4e7ddefca857-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-006b1977-czd4f\" (UID: \"ba7ec649-b8e8-4906-a1b3-4e7ddefca857\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f"
Sep 10 12:49:23.087245 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:23.087226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7ec649-b8e8-4906-a1b3-4e7ddefca857-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-006b1977-czd4f\" (UID: \"ba7ec649-b8e8-4906-a1b3-4e7ddefca857\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f"
Sep 10 12:49:23.177534 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:23.177499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f"
Sep 10 12:49:23.525501 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:23.525423    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f"]
Sep 10 12:49:23.528945 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:49:23.528915    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7ec649_b8e8_4906_a1b3_4e7ddefca857.slice/crio-6e56c11da0cce74fcee9be9b2feaa32c0b044cd90ea69b19c256926c17a72641 WatchSource:0}: Error finding container 6e56c11da0cce74fcee9be9b2feaa32c0b044cd90ea69b19c256926c17a72641: Status 404 returned error can't find the container with id 6e56c11da0cce74fcee9be9b2feaa32c0b044cd90ea69b19c256926c17a72641
Sep 10 12:49:23.795942 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:23.795850    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f" event={"ID":"ba7ec649-b8e8-4906-a1b3-4e7ddefca857","Type":"ContainerStarted","Data":"2d2915c91d15a9b36fa41cc7c8c261a62f0e84b10876015154bc1ff52352cb14"}
Sep 10 12:49:23.795942 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:23.795910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f" event={"ID":"ba7ec649-b8e8-4906-a1b3-4e7ddefca857","Type":"ContainerStarted","Data":"6e56c11da0cce74fcee9be9b2feaa32c0b044cd90ea69b19c256926c17a72641"}
Sep 10 12:49:23.841928 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:23.841876    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f" podStartSLOduration=1.8418577520000001 podStartE2EDuration="1.841857752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:49:23.840307587 +0000 UTC m=+1137.765219075" watchObservedRunningTime="2026-09-10 12:49:23.841857752 +0000 UTC m=+1137.766769241"
Sep 10 12:49:24.803647 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:24.803610    2563 generic.go:356] "Generic (PLEG): container finished" podID="ba7ec649-b8e8-4906-a1b3-4e7ddefca857" containerID="2d2915c91d15a9b36fa41cc7c8c261a62f0e84b10876015154bc1ff52352cb14" exitCode=0
Sep 10 12:49:24.804023 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:24.803703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f" event={"ID":"ba7ec649-b8e8-4906-a1b3-4e7ddefca857","Type":"ContainerDied","Data":"2d2915c91d15a9b36fa41cc7c8c261a62f0e84b10876015154bc1ff52352cb14"}
Sep 10 12:49:25.946726 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:25.946699    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f"
Sep 10 12:49:26.118575 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:26.118469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7ec649-b8e8-4906-a1b3-4e7ddefca857-maas-api-service-ca\") pod \"ba7ec649-b8e8-4906-a1b3-4e7ddefca857\" (UID: \"ba7ec649-b8e8-4906-a1b3-4e7ddefca857\") "
Sep 10 12:49:26.118845 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:26.118822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7ec649-b8e8-4906-a1b3-4e7ddefca857-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ba7ec649-b8e8-4906-a1b3-4e7ddefca857" (UID: "ba7ec649-b8e8-4906-a1b3-4e7ddefca857"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:49:26.220170 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:26.220129    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba7ec649-b8e8-4906-a1b3-4e7ddefca857-maas-api-service-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:49:26.816613 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:26.816583    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f"
Sep 10 12:49:26.816793 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:26.816580    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f" event={"ID":"ba7ec649-b8e8-4906-a1b3-4e7ddefca857","Type":"ContainerDied","Data":"6e56c11da0cce74fcee9be9b2feaa32c0b044cd90ea69b19c256926c17a72641"}
Sep 10 12:49:26.816793 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:26.816699    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e56c11da0cce74fcee9be9b2feaa32c0b044cd90ea69b19c256926c17a72641"
Sep 10 12:49:28.416872 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:28.416839    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:49:33.794544 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:33.794505    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f"]
Sep 10 12:49:33.800024 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:33.799995    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-006b1977-czd4f"]
Sep 10 12:49:34.705830 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:34.705793    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba7ec649-b8e8-4906-a1b3-4e7ddefca857" path="/var/lib/kubelet/pods/ba7ec649-b8e8-4906-a1b3-4e7ddefca857/volumes"
Sep 10 12:49:56.443869 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:49:56.443778    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:50:24.728303 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:24.728264    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"]
Sep 10 12:50:24.728805 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:24.728516    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs" podUID="a707f87e-7b63-45d9-a092-adc6f38fbfca" containerName="istio-proxy" containerID="cri-o://95cfe66b9ba0513ad876854cf710edc7d74fa7ae09e7241d644dcdfb0427873f" gracePeriod=30
Sep 10 12:50:25.036366 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:25.036269    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:50:26.259816 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.259783    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr"]
Sep 10 12:50:26.260239 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.260226    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba7ec649-b8e8-4906-a1b3-4e7ddefca857" containerName="revoke-keys"
Sep 10 12:50:26.260239 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.260240    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba7ec649-b8e8-4906-a1b3-4e7ddefca857" containerName="revoke-keys"
Sep 10 12:50:26.260334 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.260325    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba7ec649-b8e8-4906-a1b3-4e7ddefca857" containerName="revoke-keys"
Sep 10 12:50:26.263618 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.263602    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr"
Sep 10 12:50:26.267625 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.267606    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:50:26.267746 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.267726    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:50:26.275151 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.275122    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr"]
Sep 10 12:50:26.352218 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.352187    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d8e25d-433a-4971-847a-7bf588a9f8a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr\" (UID: \"04d8e25d-433a-4971-847a-7bf588a9f8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr"
Sep 10 12:50:26.453264 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.453226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d8e25d-433a-4971-847a-7bf588a9f8a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr\" (UID: \"04d8e25d-433a-4971-847a-7bf588a9f8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr"
Sep 10 12:50:26.456136 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.456115    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:50:26.463928 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.463898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d8e25d-433a-4971-847a-7bf588a9f8a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr\" (UID: \"04d8e25d-433a-4971-847a-7bf588a9f8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr"
Sep 10 12:50:26.577188 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.577094    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:50:26.584110 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.584084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr"
Sep 10 12:50:26.876147 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:26.876074    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr"]
Sep 10 12:50:26.879500 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:50:26.879469    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d8e25d_433a_4971_847a_7bf588a9f8a1.slice/crio-b671eadb08897d24694ad38e01e510a16cfb5ee420f4d323f2a4bffc8a920e45 WatchSource:0}: Error finding container b671eadb08897d24694ad38e01e510a16cfb5ee420f4d323f2a4bffc8a920e45: Status 404 returned error can't find the container with id b671eadb08897d24694ad38e01e510a16cfb5ee420f4d323f2a4bffc8a920e45
Sep 10 12:50:27.189159 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:27.189128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr" event={"ID":"04d8e25d-433a-4971-847a-7bf588a9f8a1","Type":"ContainerStarted","Data":"b671eadb08897d24694ad38e01e510a16cfb5ee420f4d323f2a4bffc8a920e45"}
Sep 10 12:50:28.200218 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:28.200184    2563 generic.go:356] "Generic (PLEG): container finished" podID="04d8e25d-433a-4971-847a-7bf588a9f8a1" containerID="1af696f8916b53e69138be40d3f3c83e9d3ac6d41f1a9f572e15f2db16245e18" exitCode=0
Sep 10 12:50:28.200651 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:28.200238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr" event={"ID":"04d8e25d-433a-4971-847a-7bf588a9f8a1","Type":"ContainerDied","Data":"1af696f8916b53e69138be40d3f3c83e9d3ac6d41f1a9f572e15f2db16245e18"}
Sep 10 12:50:29.354683 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:29.354659    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr"
Sep 10 12:50:29.484038 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:29.483961    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d8e25d-433a-4971-847a-7bf588a9f8a1-maas-api-service-ca\") pod \"04d8e25d-433a-4971-847a-7bf588a9f8a1\" (UID: \"04d8e25d-433a-4971-847a-7bf588a9f8a1\") "
Sep 10 12:50:29.484308 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:29.484283    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d8e25d-433a-4971-847a-7bf588a9f8a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04d8e25d-433a-4971-847a-7bf588a9f8a1" (UID: "04d8e25d-433a-4971-847a-7bf588a9f8a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:50:29.585709 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:29.585676    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d8e25d-433a-4971-847a-7bf588a9f8a1-maas-api-service-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:50:29.992712 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:29.992640    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:50:30.089611 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.089575    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck6kz\" (UniqueName: \"kubernetes.io/projected/a707f87e-7b63-45d9-a092-adc6f38fbfca-kube-api-access-ck6kz\") pod \"a707f87e-7b63-45d9-a092-adc6f38fbfca\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") "
Sep 10 12:50:30.089807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.089632    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-credential-socket\") pod \"a707f87e-7b63-45d9-a092-adc6f38fbfca\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") "
Sep 10 12:50:30.089807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.089651    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-workload-socket\") pod \"a707f87e-7b63-45d9-a092-adc6f38fbfca\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") "
Sep 10 12:50:30.089807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.089668    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-workload-certs\") pod \"a707f87e-7b63-45d9-a092-adc6f38fbfca\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") "
Sep 10 12:50:30.089807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.089684    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a707f87e-7b63-45d9-a092-adc6f38fbfca-istiod-ca-cert\") pod \"a707f87e-7b63-45d9-a092-adc6f38fbfca\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") "
Sep 10 12:50:30.089807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.089717    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-token\") pod \"a707f87e-7b63-45d9-a092-adc6f38fbfca\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") "
Sep 10 12:50:30.089807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.089744    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-data\") pod \"a707f87e-7b63-45d9-a092-adc6f38fbfca\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") "
Sep 10 12:50:30.090082 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.089816    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-podinfo\") pod \"a707f87e-7b63-45d9-a092-adc6f38fbfca\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") "
Sep 10 12:50:30.090082 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.089853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-envoy\") pod \"a707f87e-7b63-45d9-a092-adc6f38fbfca\" (UID: \"a707f87e-7b63-45d9-a092-adc6f38fbfca\") "
Sep 10 12:50:30.090082 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.089900    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a707f87e-7b63-45d9-a092-adc6f38fbfca" (UID: "a707f87e-7b63-45d9-a092-adc6f38fbfca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:50:30.090378 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.090342    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a707f87e-7b63-45d9-a092-adc6f38fbfca" (UID: "a707f87e-7b63-45d9-a092-adc6f38fbfca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:50:30.090472 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.090413    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a707f87e-7b63-45d9-a092-adc6f38fbfca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a707f87e-7b63-45d9-a092-adc6f38fbfca" (UID: "a707f87e-7b63-45d9-a092-adc6f38fbfca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:50:30.090597 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.090547    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-data" (OuterVolumeSpecName: "istio-data") pod "a707f87e-7b63-45d9-a092-adc6f38fbfca" (UID: "a707f87e-7b63-45d9-a092-adc6f38fbfca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:50:30.090996 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.090739    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a707f87e-7b63-45d9-a092-adc6f38fbfca" (UID: "a707f87e-7b63-45d9-a092-adc6f38fbfca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:50:30.091342 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.091321    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-credential-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:50:30.091409 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.091357    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-workload-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:50:30.091409 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.091373    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-workload-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:50:30.091409 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.091387    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a707f87e-7b63-45d9-a092-adc6f38fbfca-istiod-ca-cert\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:50:30.091409 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.091400    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-data\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:50:30.098989 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.093562    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a707f87e-7b63-45d9-a092-adc6f38fbfca" (UID: "a707f87e-7b63-45d9-a092-adc6f38fbfca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:50:30.098989 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.094466    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-token" (OuterVolumeSpecName: "istio-token") pod "a707f87e-7b63-45d9-a092-adc6f38fbfca" (UID: "a707f87e-7b63-45d9-a092-adc6f38fbfca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:50:30.099147 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.099072    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a707f87e-7b63-45d9-a092-adc6f38fbfca-kube-api-access-ck6kz" (OuterVolumeSpecName: "kube-api-access-ck6kz") pod "a707f87e-7b63-45d9-a092-adc6f38fbfca" (UID: "a707f87e-7b63-45d9-a092-adc6f38fbfca"). InnerVolumeSpecName "kube-api-access-ck6kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:50:30.100424 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.100404    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a707f87e-7b63-45d9-a092-adc6f38fbfca" (UID: "a707f87e-7b63-45d9-a092-adc6f38fbfca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:50:30.192588 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.192528    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-envoy\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:50:30.192588 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.192584    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck6kz\" (UniqueName: \"kubernetes.io/projected/a707f87e-7b63-45d9-a092-adc6f38fbfca-kube-api-access-ck6kz\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:50:30.192588 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.192595    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-token\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:50:30.192588 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.192603    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a707f87e-7b63-45d9-a092-adc6f38fbfca-istio-podinfo\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:50:30.213143 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.213108    2563 generic.go:356] "Generic (PLEG): container finished" podID="a707f87e-7b63-45d9-a092-adc6f38fbfca" containerID="95cfe66b9ba0513ad876854cf710edc7d74fa7ae09e7241d644dcdfb0427873f" exitCode=0
Sep 10 12:50:30.213295 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.213167    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"
Sep 10 12:50:30.213295 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.213190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs" event={"ID":"a707f87e-7b63-45d9-a092-adc6f38fbfca","Type":"ContainerDied","Data":"95cfe66b9ba0513ad876854cf710edc7d74fa7ae09e7241d644dcdfb0427873f"}
Sep 10 12:50:30.213295 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.213275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs" event={"ID":"a707f87e-7b63-45d9-a092-adc6f38fbfca","Type":"ContainerDied","Data":"707d0d9bc3329dddd528b20d5b0f8f58a1e2d829736a9f4afed45b539052790d"}
Sep 10 12:50:30.213431 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.213308    2563 scope.go:117] "RemoveContainer" containerID="95cfe66b9ba0513ad876854cf710edc7d74fa7ae09e7241d644dcdfb0427873f"
Sep 10 12:50:30.214718 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.214686    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr"
Sep 10 12:50:30.214784 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.214688    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr" event={"ID":"04d8e25d-433a-4971-847a-7bf588a9f8a1","Type":"ContainerDied","Data":"b671eadb08897d24694ad38e01e510a16cfb5ee420f4d323f2a4bffc8a920e45"}
Sep 10 12:50:30.214835 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.214810    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b671eadb08897d24694ad38e01e510a16cfb5ee420f4d323f2a4bffc8a920e45"
Sep 10 12:50:30.223399 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.223383    2563 scope.go:117] "RemoveContainer" containerID="95cfe66b9ba0513ad876854cf710edc7d74fa7ae09e7241d644dcdfb0427873f"
Sep 10 12:50:30.223820 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:50:30.223755    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95cfe66b9ba0513ad876854cf710edc7d74fa7ae09e7241d644dcdfb0427873f\": container with ID starting with 95cfe66b9ba0513ad876854cf710edc7d74fa7ae09e7241d644dcdfb0427873f not found: ID does not exist" containerID="95cfe66b9ba0513ad876854cf710edc7d74fa7ae09e7241d644dcdfb0427873f"
Sep 10 12:50:30.223820 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.223787    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cfe66b9ba0513ad876854cf710edc7d74fa7ae09e7241d644dcdfb0427873f"} err="failed to get container status \"95cfe66b9ba0513ad876854cf710edc7d74fa7ae09e7241d644dcdfb0427873f\": rpc error: code = NotFound desc = could not find container \"95cfe66b9ba0513ad876854cf710edc7d74fa7ae09e7241d644dcdfb0427873f\": container with ID starting with 95cfe66b9ba0513ad876854cf710edc7d74fa7ae09e7241d644dcdfb0427873f not found: ID does not exist"
Sep 10 12:50:30.245680 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.245615    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"]
Sep 10 12:50:30.261230 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.261204    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-340098-openshift-default-8dfbfddf6-mt4zs"]
Sep 10 12:50:30.706359 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:30.706324    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a707f87e-7b63-45d9-a092-adc6f38fbfca" path="/var/lib/kubelet/pods/a707f87e-7b63-45d9-a092-adc6f38fbfca/volumes"
Sep 10 12:50:31.115695 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.115610    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"]
Sep 10 12:50:31.116070 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.116056    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04d8e25d-433a-4971-847a-7bf588a9f8a1" containerName="revoke-keys"
Sep 10 12:50:31.116137 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.116070    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="04d8e25d-433a-4971-847a-7bf588a9f8a1" containerName="revoke-keys"
Sep 10 12:50:31.116137 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.116081    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a707f87e-7b63-45d9-a092-adc6f38fbfca" containerName="istio-proxy"
Sep 10 12:50:31.116137 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.116114    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a707f87e-7b63-45d9-a092-adc6f38fbfca" containerName="istio-proxy"
Sep 10 12:50:31.116293 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.116201    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a707f87e-7b63-45d9-a092-adc6f38fbfca" containerName="istio-proxy"
Sep 10 12:50:31.116293 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.116212    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="04d8e25d-433a-4971-847a-7bf588a9f8a1" containerName="revoke-keys"
Sep 10 12:50:31.121015 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.120995    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.140265 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.140240    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-719d8a-openshift-default-dockercfg-krvfv\""
Sep 10 12:50:31.236773 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.236741    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"]
Sep 10 12:50:31.305509 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.305465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-podinfo\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.305725 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.305517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-credential-socket\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.305725 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.305620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-workload-socket\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.305725 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.305653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-workload-certs\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.305725 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.305670    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-data\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.305725 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.305697    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ade2931-49ff-4226-8c5d-05199d28dc0e-istiod-ca-cert\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.305725 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.305717    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-envoy\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.305958 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.305785    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4924\" (UniqueName: \"kubernetes.io/projected/6ade2931-49ff-4226-8c5d-05199d28dc0e-kube-api-access-v4924\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.305958 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.305836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-token\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.406911 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.406820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-workload-socket\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.406911 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.406858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-workload-certs\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.406911 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.406876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-data\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.406911 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.406902    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ade2931-49ff-4226-8c5d-05199d28dc0e-istiod-ca-cert\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.407253 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.406922    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-envoy\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.407253 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.406972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4924\" (UniqueName: \"kubernetes.io/projected/6ade2931-49ff-4226-8c5d-05199d28dc0e-kube-api-access-v4924\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.407253 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.407015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-token\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.407253 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.407104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-podinfo\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.407253 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.407141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-credential-socket\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.407541 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.407291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-workload-socket\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.407665 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.407645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-data\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.407794 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.407771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-workload-certs\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.407861 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.407848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-credential-socket\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.408052 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.408031    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ade2931-49ff-4226-8c5d-05199d28dc0e-istiod-ca-cert\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.409236 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.409218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-envoy\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.409906 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.409891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-podinfo\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.421948 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.421921    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4924\" (UniqueName: \"kubernetes.io/projected/6ade2931-49ff-4226-8c5d-05199d28dc0e-kube-api-access-v4924\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.422423 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.422406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-token\") pod \"e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") " pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.431013 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.430991    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:31.748158 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.748127    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"]
Sep 10 12:50:31.750109 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:50:31.750078    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ade2931_49ff_4226_8c5d_05199d28dc0e.slice/crio-5434c7b3d98626e3385fac62eba1897323ffe8eafa9686955bdb25a95e4a3a12 WatchSource:0}: Error finding container 5434c7b3d98626e3385fac62eba1897323ffe8eafa9686955bdb25a95e4a3a12: Status 404 returned error can't find the container with id 5434c7b3d98626e3385fac62eba1897323ffe8eafa9686955bdb25a95e4a3a12
Sep 10 12:50:31.752414 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.752380    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:50:31.752509 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.752460    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:50:31.752588 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:31.752530    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:50:32.230131 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:32.230097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" event={"ID":"6ade2931-49ff-4226-8c5d-05199d28dc0e","Type":"ContainerStarted","Data":"f9a349f61718cf6428542ddd2e42f7185fc134b34f45d06df5a3c50a7fc3a85e"}
Sep 10 12:50:32.230323 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:32.230162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" event={"ID":"6ade2931-49ff-4226-8c5d-05199d28dc0e","Type":"ContainerStarted","Data":"5434c7b3d98626e3385fac62eba1897323ffe8eafa9686955bdb25a95e4a3a12"}
Sep 10 12:50:32.268474 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:32.268417    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" podStartSLOduration=1.268402785 podStartE2EDuration="1.268402785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:50:32.266659313 +0000 UTC m=+1206.191570879" watchObservedRunningTime="2026-09-10 12:50:32.268402785 +0000 UTC m=+1206.193314349"
Sep 10 12:50:32.431729 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:32.431696    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:32.433094 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:32.433067    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 10 12:50:32.433221 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:32.433125    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 10 12:50:33.435516 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:33.435482    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:34.241411 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:34.241379    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:34.242393 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:34.242372    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:50:37.209777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:37.209744    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr"]
Sep 10 12:50:37.216598 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:37.216572    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f85889-qhkpr"]
Sep 10 12:50:38.705228 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:38.705193    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04d8e25d-433a-4971-847a-7bf588a9f8a1" path="/var/lib/kubelet/pods/04d8e25d-433a-4971-847a-7bf588a9f8a1/volumes"
Sep 10 12:50:40.566934 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.566899    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"]
Sep 10 12:50:40.570672 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.570653    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.574371 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:50:40.574338    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"e2e-models-e2e-worker-w4-1312ea\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" podUID="b03a628d-daed-477f-aa0d-bdfbbb50ef7e" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.574508 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:50:40.574368    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"distinct-2-w4-1312ea-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-133-150.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-models-e2e-worker-w4-1312ea\": no relationship found between node 'ip-10-0-133-150.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-models-e2e-worker-w4-1312ea\"/\"distinct-2-w4-1312ea-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 10 12:50:40.608206 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.608175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-home\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.608329 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.608233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-tmp-dir\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.608329 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.608281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-model-cache\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.608329 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.608320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-dshm\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.608440 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.608358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-tls-certs\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.608440 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.608387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m8xq\" (UniqueName: \"kubernetes.io/projected/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-kube-api-access-5m8xq\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.631012 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.630988    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"]
Sep 10 12:50:40.709091 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.709060    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-dshm\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.709272 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.709120    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-tls-certs\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.709272 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.709168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m8xq\" (UniqueName: \"kubernetes.io/projected/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-kube-api-access-5m8xq\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.709272 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.709230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-home\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.709439 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.709300    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-tmp-dir\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.709439 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.709344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-model-cache\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.709628 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.709605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-home\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.709712 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.709691    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-model-cache\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.709772 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.709720    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-tmp-dir\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.711311 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.711292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-dshm\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:40.721685 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:40.721663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m8xq\" (UniqueName: \"kubernetes.io/projected/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-kube-api-access-5m8xq\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:41.435121 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:41.435091    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-1312ea\"/\"distinct-2-w4-1312ea-kserve-self-signed-certs\""
Sep 10 12:50:41.443169 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:41.443146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-tls-certs\") pod \"distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") " pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:41.480171 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:41.480141    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:41.764466 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:41.764442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"]
Sep 10 12:50:41.766668 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:50:41.766639    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb03a628d_daed_477f_aa0d_bdfbbb50ef7e.slice/crio-b6dda2bb01ac679e8548e971d2e0af9fb59a0393ba45e9390ff38b4106711aa1 WatchSource:0}: Error finding container b6dda2bb01ac679e8548e971d2e0af9fb59a0393ba45e9390ff38b4106711aa1: Status 404 returned error can't find the container with id b6dda2bb01ac679e8548e971d2e0af9fb59a0393ba45e9390ff38b4106711aa1
Sep 10 12:50:42.292878 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:42.292842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579" event={"ID":"b03a628d-daed-477f-aa0d-bdfbbb50ef7e","Type":"ContainerStarted","Data":"a7801efcff397bfb14b6deff56364afa3fa593ad40a4e4c432a91bf986efb4ce"}
Sep 10 12:50:42.292878 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:42.292887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579" event={"ID":"b03a628d-daed-477f-aa0d-bdfbbb50ef7e","Type":"ContainerStarted","Data":"b6dda2bb01ac679e8548e971d2e0af9fb59a0393ba45e9390ff38b4106711aa1"}
Sep 10 12:50:42.319232 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:42.319180    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579" podStartSLOduration=2.319165096 podStartE2EDuration="2.319165096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:50:42.316592441 +0000 UTC m=+1216.241503929" watchObservedRunningTime="2026-09-10 12:50:42.319165096 +0000 UTC m=+1216.244076582"
Sep 10 12:50:51.481204 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:51.481166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:51.481642 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:51.481217    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:51.494142 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:51.494117    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:52.363671 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:52.363641    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:50:54.914767 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:50:54.914726    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:51:05.282738 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:05.282697    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 12:51:05.283147 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:05.282765    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 12:51:06.440325 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:06.440291    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerID="f9a349f61718cf6428542ddd2e42f7185fc134b34f45d06df5a3c50a7fc3a85e" exitCode=0
Sep 10 12:51:06.440773 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:06.440340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" event={"ID":"6ade2931-49ff-4226-8c5d-05199d28dc0e","Type":"ContainerDied","Data":"f9a349f61718cf6428542ddd2e42f7185fc134b34f45d06df5a3c50a7fc3a85e"}
Sep 10 12:51:06.440773 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:06.440732    2563 scope.go:117] "RemoveContainer" containerID="f9a349f61718cf6428542ddd2e42f7185fc134b34f45d06df5a3c50a7fc3a85e"
Sep 10 12:51:06.441906 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:06.441878    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:51:06.441984 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:06.441948    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:51:06.442024 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:06.441982    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:51:07.450800 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:07.450757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" event={"ID":"6ade2931-49ff-4226-8c5d-05199d28dc0e","Type":"ContainerStarted","Data":"b481bbf836043fb5921a5706634c1254124d2e97ffa6a1dd55a62dff2d88fee1"}
Sep 10 12:51:08.432133 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:08.432096    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:51:08.432386 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:08.432360    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 10 12:51:08.432432 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:08.432415    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 10 12:51:09.432300 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:09.432264    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 10 12:51:09.432708 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:09.432320    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 10 12:51:10.018770 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:10.018739    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"]
Sep 10 12:51:10.019002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:10.018959    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv" podUID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerName="istio-proxy" containerID="cri-o://07af6887d518edb463e25e6f6b12e2bd5deb6c6a3086a8f23f0556dbb21daa3b" gracePeriod=30
Sep 10 12:51:10.432012 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:10.431983    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 10 12:51:10.432171 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:10.432044    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 10 12:51:11.431354 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:11.431316    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Sep 10 12:51:11.431770 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:11.431385    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Sep 10 12:51:12.443912 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:12.443873    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:51:12.444345 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:12.444305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:51:12.445185 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:12.445161    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:51:15.282567 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.282498    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:51:15.395644 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.395615    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5lwd\" (UniqueName: \"kubernetes.io/projected/4d4af67b-c9b1-4075-a28e-c7151d5094a2-kube-api-access-q5lwd\") pod \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") "
Sep 10 12:51:15.395834 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.395669    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-token\") pod \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") "
Sep 10 12:51:15.395834 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.395735    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-podinfo\") pod \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") "
Sep 10 12:51:15.395834 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.395762    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istiod-ca-cert\") pod \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") "
Sep 10 12:51:15.395834 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.395808    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-credential-socket\") pod \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") "
Sep 10 12:51:15.395834 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.395831    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-data\") pod \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") "
Sep 10 12:51:15.396090 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.395852    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-workload-certs\") pod \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") "
Sep 10 12:51:15.396090 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.395888    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-workload-socket\") pod \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") "
Sep 10 12:51:15.396090 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.395937    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-envoy\") pod \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\" (UID: \"4d4af67b-c9b1-4075-a28e-c7151d5094a2\") "
Sep 10 12:51:15.396253 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.396079    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d4af67b-c9b1-4075-a28e-c7151d5094a2" (UID: "4d4af67b-c9b1-4075-a28e-c7151d5094a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:51:15.396253 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.396094    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d4af67b-c9b1-4075-a28e-c7151d5094a2" (UID: "4d4af67b-c9b1-4075-a28e-c7151d5094a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:51:15.396253 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.396155    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d4af67b-c9b1-4075-a28e-c7151d5094a2" (UID: "4d4af67b-c9b1-4075-a28e-c7151d5094a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:51:15.396462 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.396266    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d4af67b-c9b1-4075-a28e-c7151d5094a2" (UID: "4d4af67b-c9b1-4075-a28e-c7151d5094a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:51:15.396462 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.396298    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d4af67b-c9b1-4075-a28e-c7151d5094a2" (UID: "4d4af67b-c9b1-4075-a28e-c7151d5094a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:51:15.396462 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.396426    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istiod-ca-cert\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:51:15.396462 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.396448    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-credential-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:51:15.396462 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.396462    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-data\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:51:15.396759 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.396475    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-workload-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:51:15.396759 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.396487    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-workload-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:51:15.398043 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.398007    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4af67b-c9b1-4075-a28e-c7151d5094a2-kube-api-access-q5lwd" (OuterVolumeSpecName: "kube-api-access-q5lwd") pod "4d4af67b-c9b1-4075-a28e-c7151d5094a2" (UID: "4d4af67b-c9b1-4075-a28e-c7151d5094a2"). InnerVolumeSpecName "kube-api-access-q5lwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:51:15.398043 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.398025    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d4af67b-c9b1-4075-a28e-c7151d5094a2" (UID: "4d4af67b-c9b1-4075-a28e-c7151d5094a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:51:15.398280 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.398088    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d4af67b-c9b1-4075-a28e-c7151d5094a2" (UID: "4d4af67b-c9b1-4075-a28e-c7151d5094a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:51:15.398280 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.398218    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d4af67b-c9b1-4075-a28e-c7151d5094a2" (UID: "4d4af67b-c9b1-4075-a28e-c7151d5094a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:51:15.497413 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.497377    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-envoy\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:51:15.497413 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.497416    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5lwd\" (UniqueName: \"kubernetes.io/projected/4d4af67b-c9b1-4075-a28e-c7151d5094a2-kube-api-access-q5lwd\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:51:15.497616 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.497431    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-token\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:51:15.497616 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.497445    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d4af67b-c9b1-4075-a28e-c7151d5094a2-istio-podinfo\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:51:15.511624 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.511594    2563 generic.go:356] "Generic (PLEG): container finished" podID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerID="07af6887d518edb463e25e6f6b12e2bd5deb6c6a3086a8f23f0556dbb21daa3b" exitCode=0
Sep 10 12:51:15.511762 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.511676    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"
Sep 10 12:51:15.511762 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.511708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv" event={"ID":"4d4af67b-c9b1-4075-a28e-c7151d5094a2","Type":"ContainerDied","Data":"07af6887d518edb463e25e6f6b12e2bd5deb6c6a3086a8f23f0556dbb21daa3b"}
Sep 10 12:51:15.511852 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.511764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv" event={"ID":"4d4af67b-c9b1-4075-a28e-c7151d5094a2","Type":"ContainerDied","Data":"0355c1196c772c7ce77282419cf469421202ffad3f711a9f67fec5869ad71fc4"}
Sep 10 12:51:15.511852 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.511787    2563 scope.go:117] "RemoveContainer" containerID="07af6887d518edb463e25e6f6b12e2bd5deb6c6a3086a8f23f0556dbb21daa3b"
Sep 10 12:51:15.522374 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.522355    2563 scope.go:117] "RemoveContainer" containerID="d13bc78fa9c5e5cf746dec3335d10a929dda305f2b1fc1b41a8636405cbad05d"
Sep 10 12:51:15.531702 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.531683    2563 scope.go:117] "RemoveContainer" containerID="07af6887d518edb463e25e6f6b12e2bd5deb6c6a3086a8f23f0556dbb21daa3b"
Sep 10 12:51:15.531974 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:51:15.531955    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07af6887d518edb463e25e6f6b12e2bd5deb6c6a3086a8f23f0556dbb21daa3b\": container with ID starting with 07af6887d518edb463e25e6f6b12e2bd5deb6c6a3086a8f23f0556dbb21daa3b not found: ID does not exist" containerID="07af6887d518edb463e25e6f6b12e2bd5deb6c6a3086a8f23f0556dbb21daa3b"
Sep 10 12:51:15.532039 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.531984    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07af6887d518edb463e25e6f6b12e2bd5deb6c6a3086a8f23f0556dbb21daa3b"} err="failed to get container status \"07af6887d518edb463e25e6f6b12e2bd5deb6c6a3086a8f23f0556dbb21daa3b\": rpc error: code = NotFound desc = could not find container \"07af6887d518edb463e25e6f6b12e2bd5deb6c6a3086a8f23f0556dbb21daa3b\": container with ID starting with 07af6887d518edb463e25e6f6b12e2bd5deb6c6a3086a8f23f0556dbb21daa3b not found: ID does not exist"
Sep 10 12:51:15.532039 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.532000    2563 scope.go:117] "RemoveContainer" containerID="d13bc78fa9c5e5cf746dec3335d10a929dda305f2b1fc1b41a8636405cbad05d"
Sep 10 12:51:15.532238 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:51:15.532224    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13bc78fa9c5e5cf746dec3335d10a929dda305f2b1fc1b41a8636405cbad05d\": container with ID starting with d13bc78fa9c5e5cf746dec3335d10a929dda305f2b1fc1b41a8636405cbad05d not found: ID does not exist" containerID="d13bc78fa9c5e5cf746dec3335d10a929dda305f2b1fc1b41a8636405cbad05d"
Sep 10 12:51:15.532277 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.532241    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13bc78fa9c5e5cf746dec3335d10a929dda305f2b1fc1b41a8636405cbad05d"} err="failed to get container status \"d13bc78fa9c5e5cf746dec3335d10a929dda305f2b1fc1b41a8636405cbad05d\": rpc error: code = NotFound desc = could not find container \"d13bc78fa9c5e5cf746dec3335d10a929dda305f2b1fc1b41a8636405cbad05d\": container with ID starting with d13bc78fa9c5e5cf746dec3335d10a929dda305f2b1fc1b41a8636405cbad05d not found: ID does not exist"
Sep 10 12:51:15.536135 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.536070    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"]
Sep 10 12:51:15.539092 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.539069    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-714b74-openshift-default-687644b978-9bsvv"]
Sep 10 12:51:15.859283 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.859059    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"]
Sep 10 12:51:15.859808 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.859783    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerName="istio-proxy"
Sep 10 12:51:15.859808 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.859807    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerName="istio-proxy"
Sep 10 12:51:15.860002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.859850    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerName="istio-proxy"
Sep 10 12:51:15.860002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.859861    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerName="istio-proxy"
Sep 10 12:51:15.860002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.859953    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerName="istio-proxy"
Sep 10 12:51:15.860002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.859967    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" containerName="istio-proxy"
Sep 10 12:51:15.863959 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.863941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:15.866632 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.866598    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 12:51:15.867181 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.866760    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 12:51:15.867181 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.866956    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fce322fe-serving-cert\""
Sep 10 12:51:15.867181 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.867040    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xshvt\""
Sep 10 12:51:15.867181 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.867138    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:51:15.873917 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:15.873889    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"]
Sep 10 12:51:16.003657 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.003625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-metrics-certs\") pod \"maas-api-e2e-mt-fce322fe-84756dc964-lx8tq\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:16.003834 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.003673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-maas-api-tls\") pod \"maas-api-e2e-mt-fce322fe-84756dc964-lx8tq\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:16.003834 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.003760    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29xqk\" (UniqueName: \"kubernetes.io/projected/50bde372-ac63-42be-b27c-c87a7a3f8d6d-kube-api-access-29xqk\") pod \"maas-api-e2e-mt-fce322fe-84756dc964-lx8tq\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:16.003834 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.003813    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50bde372-ac63-42be-b27c-c87a7a3f8d6d-openshift-service-ca\") pod \"maas-api-e2e-mt-fce322fe-84756dc964-lx8tq\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:16.104996 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.104936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50bde372-ac63-42be-b27c-c87a7a3f8d6d-openshift-service-ca\") pod \"maas-api-e2e-mt-fce322fe-84756dc964-lx8tq\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:16.105192 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.105067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-metrics-certs\") pod \"maas-api-e2e-mt-fce322fe-84756dc964-lx8tq\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:16.105192 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.105100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-maas-api-tls\") pod \"maas-api-e2e-mt-fce322fe-84756dc964-lx8tq\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:16.105192 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.105140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29xqk\" (UniqueName: \"kubernetes.io/projected/50bde372-ac63-42be-b27c-c87a7a3f8d6d-kube-api-access-29xqk\") pod \"maas-api-e2e-mt-fce322fe-84756dc964-lx8tq\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:16.105367 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:51:16.105245    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-serving-cert: secret "maas-api-e2e-mt-fce322fe-serving-cert" not found
Sep 10 12:51:16.105367 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:51:16.105332    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-maas-api-tls podName:50bde372-ac63-42be-b27c-c87a7a3f8d6d nodeName:}" failed. No retries permitted until 2026-09-10 12:51:16.605308455 +0000 UTC m=+1250.530219938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-maas-api-tls") pod "maas-api-e2e-mt-fce322fe-84756dc964-lx8tq" (UID: "50bde372-ac63-42be-b27c-c87a7a3f8d6d") : secret "maas-api-e2e-mt-fce322fe-serving-cert" not found
Sep 10 12:51:16.105627 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.105603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50bde372-ac63-42be-b27c-c87a7a3f8d6d-openshift-service-ca\") pod \"maas-api-e2e-mt-fce322fe-84756dc964-lx8tq\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:16.107596 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.107576    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-metrics-certs\") pod \"maas-api-e2e-mt-fce322fe-84756dc964-lx8tq\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:16.115104 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.115044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29xqk\" (UniqueName: \"kubernetes.io/projected/50bde372-ac63-42be-b27c-c87a7a3f8d6d-kube-api-access-29xqk\") pod \"maas-api-e2e-mt-fce322fe-84756dc964-lx8tq\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:16.610401 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.610360    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-maas-api-tls\") pod \"maas-api-e2e-mt-fce322fe-84756dc964-lx8tq\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:16.612933 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.612909    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-maas-api-tls\") pod \"maas-api-e2e-mt-fce322fe-84756dc964-lx8tq\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:16.706951 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.706914    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d4af67b-c9b1-4075-a28e-c7151d5094a2" path="/var/lib/kubelet/pods/4d4af67b-c9b1-4075-a28e-c7151d5094a2/volumes"
Sep 10 12:51:16.777139 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:16.777097    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:17.079637 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:17.079609    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"]
Sep 10 12:51:17.081620 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:51:17.081592    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bde372_ac63_42be_b27c_c87a7a3f8d6d.slice/crio-b3685981196f212f170f220bec97302fd4c99a85c0f7a562e702ebfdfd32dd5c WatchSource:0}: Error finding container b3685981196f212f170f220bec97302fd4c99a85c0f7a562e702ebfdfd32dd5c: Status 404 returned error can't find the container with id b3685981196f212f170f220bec97302fd4c99a85c0f7a562e702ebfdfd32dd5c
Sep 10 12:51:17.082903 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:17.082883    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:51:17.526746 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:17.526712    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq" event={"ID":"50bde372-ac63-42be-b27c-c87a7a3f8d6d","Type":"ContainerStarted","Data":"b3685981196f212f170f220bec97302fd4c99a85c0f7a562e702ebfdfd32dd5c"}
Sep 10 12:51:19.540486 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:19.540449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq" event={"ID":"50bde372-ac63-42be-b27c-c87a7a3f8d6d","Type":"ContainerStarted","Data":"7c88b33a17df25c697325ca95878842a0f79b411d9d367a8c6beb693b3925b8f"}
Sep 10 12:51:19.540848 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:19.540526    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:19.562492 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:19.562444    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq" podStartSLOduration=2.242005241 podStartE2EDuration="4.562430572s" totalImagesPullingTime="2.320425331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:51:19.561169801 +0000 UTC m=+1253.486081290" watchObservedRunningTime="2026-09-10 12:51:19.562430572 +0000 UTC m=+1253.487342058"
Sep 10 12:51:25.554635 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:25.554605    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:51:26.666727 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:26.666696    2563 scope.go:117] "RemoveContainer" containerID="5d53b9d27e3fa1c8b28b37904e3a94691091ca770525217228e28336ae05ec81"
Sep 10 12:51:26.675229 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:26.675209    2563 scope.go:117] "RemoveContainer" containerID="487eaaf723b1ffbe249555db09d2c3bd3e1cf520773e7dcf14183b5ee8e3ca10"
Sep 10 12:51:32.341363 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:32.341327    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:51:39.204090 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.204056    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"]
Sep 10 12:51:39.208449 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.208427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.211089 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.211066    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-e5208b-openshift-default-dockercfg-75gvv\""
Sep 10 12:51:39.223316 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.223290    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"]
Sep 10 12:51:39.245757 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.245724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-workload-socket\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.245904 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.245772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-token\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.245904 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.245798    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-workload-certs\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.245904 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.245839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-credential-socket\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.245904 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.245867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2766220e-dd8b-4305-b025-e1799a9a84e9-istiod-ca-cert\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.246115 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.245993    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-envoy\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.246263 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.246240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-podinfo\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.246452 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.246429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltsgw\" (UniqueName: \"kubernetes.io/projected/2766220e-dd8b-4305-b025-e1799a9a84e9-kube-api-access-ltsgw\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.246583 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.246546    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-data\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.347641 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.347604    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-credential-socket\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.347851 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.347656    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2766220e-dd8b-4305-b025-e1799a9a84e9-istiod-ca-cert\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.347851 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.347735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-envoy\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.347851 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.347772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-podinfo\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.347851 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.347812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltsgw\" (UniqueName: \"kubernetes.io/projected/2766220e-dd8b-4305-b025-e1799a9a84e9-kube-api-access-ltsgw\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.348088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.347892    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-data\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.348088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.347944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-workload-socket\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.348088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.347969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-token\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.348088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.347996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-workload-certs\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.348294 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.348167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-credential-socket\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.348348 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.348335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-workload-certs\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.348826 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.348500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2766220e-dd8b-4305-b025-e1799a9a84e9-istiod-ca-cert\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.348972 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.348873    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-workload-socket\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.349062 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.349009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-data\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.350851 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.350828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-envoy\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.351279 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.351257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-podinfo\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.358499 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.358474    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltsgw\" (UniqueName: \"kubernetes.io/projected/2766220e-dd8b-4305-b025-e1799a9a84e9-kube-api-access-ltsgw\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.360180 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.359708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-token\") pod \"e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") " pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.520334 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.520240    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:39.809452 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.809425    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"]
Sep 10 12:51:39.810897 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:51:39.810870    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2766220e_dd8b_4305_b025_e1799a9a84e9.slice/crio-aecf850c105a9a0bd5f64aad46027c69a6fd23c4f5a35aacb25e8101519ceb7f WatchSource:0}: Error finding container aecf850c105a9a0bd5f64aad46027c69a6fd23c4f5a35aacb25e8101519ceb7f: Status 404 returned error can't find the container with id aecf850c105a9a0bd5f64aad46027c69a6fd23c4f5a35aacb25e8101519ceb7f
Sep 10 12:51:39.813439 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.813400    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:51:39.813526 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.813508    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:51:39.813591 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:39.813575    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:51:40.667890 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:40.667857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" event={"ID":"2766220e-dd8b-4305-b025-e1799a9a84e9","Type":"ContainerStarted","Data":"bb5196a793fba35be4a009b2a5120a7bc6a4ce77215bd6182a3d833fd2da17e8"}
Sep 10 12:51:40.668305 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:40.667908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" event={"ID":"2766220e-dd8b-4305-b025-e1799a9a84e9","Type":"ContainerStarted","Data":"aecf850c105a9a0bd5f64aad46027c69a6fd23c4f5a35aacb25e8101519ceb7f"}
Sep 10 12:51:40.703433 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:40.703389    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" podStartSLOduration=1.703373064 podStartE2EDuration="1.703373064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:51:40.702615151 +0000 UTC m=+1274.627526656" watchObservedRunningTime="2026-09-10 12:51:40.703373064 +0000 UTC m=+1274.628284551"
Sep 10 12:51:41.520891 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:41.520857    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:41.526099 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:41.526073    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:41.680258 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:41.680219    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:41.681113 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:41.681092    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:51:43.817199 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:43.817168    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"]
Sep 10 12:51:43.820209 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:43.820190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:43.822674 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:43.822637    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e5208b-serving-cert\""
Sep 10 12:51:43.835490 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:43.835466    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"]
Sep 10 12:51:43.901751 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:43.901716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70392a30-b812-4ec2-b7ca-c8fac767edc7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:43.901939 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:43.901777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxw2b\" (UniqueName: \"kubernetes.io/projected/70392a30-b812-4ec2-b7ca-c8fac767edc7-kube-api-access-vxw2b\") pod \"maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:43.901939 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:43.901804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:43.901939 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:43.901869    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:44.002647 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:44.002611    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70392a30-b812-4ec2-b7ca-c8fac767edc7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:44.002837 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:44.002671    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxw2b\" (UniqueName: \"kubernetes.io/projected/70392a30-b812-4ec2-b7ca-c8fac767edc7-kube-api-access-vxw2b\") pod \"maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:44.002837 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:44.002701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:44.002837 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:44.002747    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:44.002998 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:51:44.002848    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-serving-cert: secret "maas-api-e2e-shared-b-w3-e5208b-serving-cert" not found
Sep 10 12:51:44.002998 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:51:44.002913    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-maas-api-tls podName:70392a30-b812-4ec2-b7ca-c8fac767edc7 nodeName:}" failed. No retries permitted until 2026-09-10 12:51:44.502897069 +0000 UTC m=+1278.427808537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-maas-api-tls") pod "maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9" (UID: "70392a30-b812-4ec2-b7ca-c8fac767edc7") : secret "maas-api-e2e-shared-b-w3-e5208b-serving-cert" not found
Sep 10 12:51:44.003340 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:44.003321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70392a30-b812-4ec2-b7ca-c8fac767edc7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:44.005170 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:44.005148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:44.014495 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:44.014474    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxw2b\" (UniqueName: \"kubernetes.io/projected/70392a30-b812-4ec2-b7ca-c8fac767edc7-kube-api-access-vxw2b\") pod \"maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:44.507540 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:44.507505    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:44.509799 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:44.509782    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:44.734481 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:44.734442    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:45.050606 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:45.050571    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"]
Sep 10 12:51:45.052493 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:51:45.052461    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70392a30_b812_4ec2_b7ca_c8fac767edc7.slice/crio-7faddb10424bfd443978284a70f2f3b4d5f6085e4aadd246c67fbe9c42f5d66f WatchSource:0}: Error finding container 7faddb10424bfd443978284a70f2f3b4d5f6085e4aadd246c67fbe9c42f5d66f: Status 404 returned error can't find the container with id 7faddb10424bfd443978284a70f2f3b4d5f6085e4aadd246c67fbe9c42f5d66f
Sep 10 12:51:45.706542 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:45.706507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9" event={"ID":"70392a30-b812-4ec2-b7ca-c8fac767edc7","Type":"ContainerStarted","Data":"beed0f69f2a7ce4d8545c78fed4507c5311a056a1f005ad4499cb3cbe0b1c01e"}
Sep 10 12:51:45.706690 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:45.706585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9" event={"ID":"70392a30-b812-4ec2-b7ca-c8fac767edc7","Type":"ContainerStarted","Data":"7faddb10424bfd443978284a70f2f3b4d5f6085e4aadd246c67fbe9c42f5d66f"}
Sep 10 12:51:45.706749 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:45.706711    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:51:45.738834 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:45.738778    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9" podStartSLOduration=2.168499508 podStartE2EDuration="2.73875771s" totalImagesPullingTime="570.258202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:51:45.736382822 +0000 UTC m=+1279.661294308" watchObservedRunningTime="2026-09-10 12:51:45.73875771 +0000 UTC m=+1279.663669200"
Sep 10 12:51:51.717252 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:51:51.717219    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 12:52:04.829614 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:04.829576    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:52:15.908842 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:15.908803    2563 generic.go:356] "Generic (PLEG): container finished" podID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerID="bb5196a793fba35be4a009b2a5120a7bc6a4ce77215bd6182a3d833fd2da17e8" exitCode=0
Sep 10 12:52:15.909263 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:15.908862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" event={"ID":"2766220e-dd8b-4305-b025-e1799a9a84e9","Type":"ContainerDied","Data":"bb5196a793fba35be4a009b2a5120a7bc6a4ce77215bd6182a3d833fd2da17e8"}
Sep 10 12:52:15.909305 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:15.909278    2563 scope.go:117] "RemoveContainer" containerID="bb5196a793fba35be4a009b2a5120a7bc6a4ce77215bd6182a3d833fd2da17e8"
Sep 10 12:52:15.910443 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:15.910414    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:52:15.910512 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:15.910483    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:52:15.910547 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:15.910517    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 12:52:16.916516 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:16.916480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" event={"ID":"2766220e-dd8b-4305-b025-e1799a9a84e9","Type":"ContainerStarted","Data":"c31c636da0a6ccaef30f46828a0a3eb1e3b48114be3f3f9a6ef66b0740d411d2"}
Sep 10 12:52:17.521075 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:17.521021    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:52:17.521316 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:17.521293    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 12:52:17.521398 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:17.521363    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 12:52:18.520735 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:18.520694    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 12:52:18.521148 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:18.520761    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 12:52:19.521319 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:19.521286    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 12:52:19.521718 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:19.521342    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 12:52:20.521232 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:20.521192    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 12:52:20.521410 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:20.521253    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 12:52:21.521491 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:21.521451    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 12:52:21.521886 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:21.521509    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 12:52:22.520785 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:22.520745    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Sep 10 12:52:22.521018 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:22.520802    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Sep 10 12:52:23.525135 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:23.525099    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:52:23.525518 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:23.525498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:52:23.526188 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:23.526171    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 12:52:26.707730 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:26.707700    2563 scope.go:117] "RemoveContainer" containerID="cfbf3ff9b89ead52c7eb9210d0434254ca8a3035380031b0a8e2659b74c4452b"
Sep 10 12:52:26.716639 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:26.716611    2563 scope.go:117] "RemoveContainer" containerID="b009b94bb03d72f1b044e727cf5c02dede5b4964eec9f114e82a5e25cce952b9"
Sep 10 12:52:41.687357 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:52:41.687316    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:53:02.813235 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:02.813151    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"]
Sep 10 12:53:02.816006 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:02.815989    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:02.822859 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:02.822486    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-30919a\"/\"unconfigured-w0-30919a-kserve-self-signed-certs\""
Sep 10 12:53:02.833308 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:02.833267    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"]
Sep 10 12:53:02.980409 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:02.980370    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-dshm\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:02.980640 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:02.980433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsjq9\" (UniqueName: \"kubernetes.io/projected/02441145-7b72-42f1-afba-7f87b9d3fd54-kube-api-access-gsjq9\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:02.980640 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:02.980491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02441145-7b72-42f1-afba-7f87b9d3fd54-tls-certs\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:02.980640 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:02.980510    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-tmp-dir\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:02.980640 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:02.980611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-home\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:02.980640 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:02.980634    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-model-cache\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.081522 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.081410    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-dshm\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.081522 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.081515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsjq9\" (UniqueName: \"kubernetes.io/projected/02441145-7b72-42f1-afba-7f87b9d3fd54-kube-api-access-gsjq9\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.081800 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.081578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02441145-7b72-42f1-afba-7f87b9d3fd54-tls-certs\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.081800 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.081609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-tmp-dir\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.081800 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.081684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-home\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.081800 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.081706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-model-cache\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.082088 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.082054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-tmp-dir\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.082287 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.082094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-home\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.082287 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.082139    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-model-cache\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.084438 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.084403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-dshm\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.084837 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.084820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02441145-7b72-42f1-afba-7f87b9d3fd54-tls-certs\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.092537 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.092506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsjq9\" (UniqueName: \"kubernetes.io/projected/02441145-7b72-42f1-afba-7f87b9d3fd54-kube-api-access-gsjq9\") pod \"unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") " pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.125509 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.125472    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:03.402865 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:03.402837    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"]
Sep 10 12:53:03.405009 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:53:03.404963    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02441145_7b72_42f1_afba_7f87b9d3fd54.slice/crio-d66d6dd52fdb8adb15ffbaee7568e6a793b8da5c4117201e41fca8c1d05ebb02 WatchSource:0}: Error finding container d66d6dd52fdb8adb15ffbaee7568e6a793b8da5c4117201e41fca8c1d05ebb02: Status 404 returned error can't find the container with id d66d6dd52fdb8adb15ffbaee7568e6a793b8da5c4117201e41fca8c1d05ebb02
Sep 10 12:53:04.223641 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:04.223598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm" event={"ID":"02441145-7b72-42f1-afba-7f87b9d3fd54","Type":"ContainerStarted","Data":"12b31ae2ffd3b7865a608f2ec74ebb198984f5faae1aa86e2d22cbb153d2f750"}
Sep 10 12:53:04.224026 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:04.223653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm" event={"ID":"02441145-7b72-42f1-afba-7f87b9d3fd54","Type":"ContainerStarted","Data":"d66d6dd52fdb8adb15ffbaee7568e6a793b8da5c4117201e41fca8c1d05ebb02"}
Sep 10 12:53:04.244996 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:04.244854    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm" podStartSLOduration=2.244836098 podStartE2EDuration="2.244836098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:53:04.242478401 +0000 UTC m=+1358.167389912" watchObservedRunningTime="2026-09-10 12:53:04.244836098 +0000 UTC m=+1358.169747584"
Sep 10 12:53:12.830120 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:12.830083    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:53:13.126572 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:13.126461    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:13.126734 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:13.126608    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:13.139600 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:13.139544    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:13.293538 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:13.293506    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:53:26.741648 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:26.741612    2563 scope.go:117] "RemoveContainer" containerID="b583e6a531c73e87aa6415268f3a1d388dbd320722d63de823583db20218a97a"
Sep 10 12:53:55.542068 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:55.542034    2563 generic.go:356] "Generic (PLEG): container finished" podID="ce40549a-16c6-4438-995e-01f1f68546da" containerID="0abdb3fce5bb3fc57c07d5f8832ee13efb596310f7c7dc96322a02e0fa5e1900" exitCode=137
Sep 10 12:53:55.542493 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:55.542115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t" event={"ID":"ce40549a-16c6-4438-995e-01f1f68546da","Type":"ContainerDied","Data":"0abdb3fce5bb3fc57c07d5f8832ee13efb596310f7c7dc96322a02e0fa5e1900"}
Sep 10 12:53:55.542493 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:55.542477    2563 scope.go:117] "RemoveContainer" containerID="0abdb3fce5bb3fc57c07d5f8832ee13efb596310f7c7dc96322a02e0fa5e1900"
Sep 10 12:53:56.551616 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:56.551578    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t" event={"ID":"ce40549a-16c6-4438-995e-01f1f68546da","Type":"ContainerStarted","Data":"e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b"}
Sep 10 12:53:56.552062 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:53:56.551814    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:54:02.321261 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.321227    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"]
Sep 10 12:54:02.321683 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.321467    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq" podUID="50bde372-ac63-42be-b27c-c87a7a3f8d6d" containerName="maas-api" containerID="cri-o://7c88b33a17df25c697325ca95878842a0f79b411d9d367a8c6beb693b3925b8f" gracePeriod=30
Sep 10 12:54:02.582968 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.582892    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:54:02.590245 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.590213    2563 generic.go:356] "Generic (PLEG): container finished" podID="50bde372-ac63-42be-b27c-c87a7a3f8d6d" containerID="7c88b33a17df25c697325ca95878842a0f79b411d9d367a8c6beb693b3925b8f" exitCode=0
Sep 10 12:54:02.590425 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.590256    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"
Sep 10 12:54:02.590425 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.590300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq" event={"ID":"50bde372-ac63-42be-b27c-c87a7a3f8d6d","Type":"ContainerDied","Data":"7c88b33a17df25c697325ca95878842a0f79b411d9d367a8c6beb693b3925b8f"}
Sep 10 12:54:02.590425 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.590353    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq" event={"ID":"50bde372-ac63-42be-b27c-c87a7a3f8d6d","Type":"ContainerDied","Data":"b3685981196f212f170f220bec97302fd4c99a85c0f7a562e702ebfdfd32dd5c"}
Sep 10 12:54:02.590425 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.590378    2563 scope.go:117] "RemoveContainer" containerID="7c88b33a17df25c697325ca95878842a0f79b411d9d367a8c6beb693b3925b8f"
Sep 10 12:54:02.599412 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.599392    2563 scope.go:117] "RemoveContainer" containerID="7c88b33a17df25c697325ca95878842a0f79b411d9d367a8c6beb693b3925b8f"
Sep 10 12:54:02.599808 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:54:02.599786    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c88b33a17df25c697325ca95878842a0f79b411d9d367a8c6beb693b3925b8f\": container with ID starting with 7c88b33a17df25c697325ca95878842a0f79b411d9d367a8c6beb693b3925b8f not found: ID does not exist" containerID="7c88b33a17df25c697325ca95878842a0f79b411d9d367a8c6beb693b3925b8f"
Sep 10 12:54:02.599874 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.599819    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c88b33a17df25c697325ca95878842a0f79b411d9d367a8c6beb693b3925b8f"} err="failed to get container status \"7c88b33a17df25c697325ca95878842a0f79b411d9d367a8c6beb693b3925b8f\": rpc error: code = NotFound desc = could not find container \"7c88b33a17df25c697325ca95878842a0f79b411d9d367a8c6beb693b3925b8f\": container with ID starting with 7c88b33a17df25c697325ca95878842a0f79b411d9d367a8c6beb693b3925b8f not found: ID does not exist"
Sep 10 12:54:02.650953 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.650923    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50bde372-ac63-42be-b27c-c87a7a3f8d6d-openshift-service-ca\") pod \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") "
Sep 10 12:54:02.651125 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.651009    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29xqk\" (UniqueName: \"kubernetes.io/projected/50bde372-ac63-42be-b27c-c87a7a3f8d6d-kube-api-access-29xqk\") pod \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") "
Sep 10 12:54:02.651125 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.651065    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-maas-api-tls\") pod \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") "
Sep 10 12:54:02.651125 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.651097    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-metrics-certs\") pod \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\" (UID: \"50bde372-ac63-42be-b27c-c87a7a3f8d6d\") "
Sep 10 12:54:02.651372 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.651344    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50bde372-ac63-42be-b27c-c87a7a3f8d6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "50bde372-ac63-42be-b27c-c87a7a3f8d6d" (UID: "50bde372-ac63-42be-b27c-c87a7a3f8d6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:54:02.651470 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.651453    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50bde372-ac63-42be-b27c-c87a7a3f8d6d-openshift-service-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:54:02.653274 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.653240    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "50bde372-ac63-42be-b27c-c87a7a3f8d6d" (UID: "50bde372-ac63-42be-b27c-c87a7a3f8d6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:54:02.653403 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.653287    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50bde372-ac63-42be-b27c-c87a7a3f8d6d-kube-api-access-29xqk" (OuterVolumeSpecName: "kube-api-access-29xqk") pod "50bde372-ac63-42be-b27c-c87a7a3f8d6d" (UID: "50bde372-ac63-42be-b27c-c87a7a3f8d6d"). InnerVolumeSpecName "kube-api-access-29xqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:54:02.653403 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.653288    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "50bde372-ac63-42be-b27c-c87a7a3f8d6d" (UID: "50bde372-ac63-42be-b27c-c87a7a3f8d6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:54:02.752216 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.752186    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-maas-api-tls\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:54:02.752216 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.752215    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50bde372-ac63-42be-b27c-c87a7a3f8d6d-metrics-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:54:02.752414 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.752228    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29xqk\" (UniqueName: \"kubernetes.io/projected/50bde372-ac63-42be-b27c-c87a7a3f8d6d-kube-api-access-29xqk\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:54:02.910179 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.910088    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"]
Sep 10 12:54:02.915358 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:02.915329    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fce322fe-84756dc964-lx8tq"]
Sep 10 12:54:04.708223 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:04.708180    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50bde372-ac63-42be-b27c-c87a7a3f8d6d" path="/var/lib/kubelet/pods/50bde372-ac63-42be-b27c-c87a7a3f8d6d/volumes"
Sep 10 12:54:07.559951 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:07.559922    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:54:26.669488 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:26.669402    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:54:26.766146 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:54:26.766110    2563 scope.go:117] "RemoveContainer" containerID="fbc1c7964774422d54ad8cc5a04a10a684df559901aa1ba05ddc13f2f07f7385"
Sep 10 12:55:00.380164 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:55:00.380119    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:55:26.802923 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:55:26.802883    2563 scope.go:117] "RemoveContainer" containerID="2d2915c91d15a9b36fa41cc7c8c261a62f0e84b10876015154bc1ff52352cb14"
Sep 10 12:55:54.556574 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:55:54.556455    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:56:18.440916 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:18.440873    2563 generic.go:356] "Generic (PLEG): container finished" podID="ce40549a-16c6-4438-995e-01f1f68546da" containerID="e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b" exitCode=137
Sep 10 12:56:18.441369 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:18.440950    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t" event={"ID":"ce40549a-16c6-4438-995e-01f1f68546da","Type":"ContainerDied","Data":"e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b"}
Sep 10 12:56:18.441369 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:18.441014    2563 scope.go:117] "RemoveContainer" containerID="0abdb3fce5bb3fc57c07d5f8832ee13efb596310f7c7dc96322a02e0fa5e1900"
Sep 10 12:56:18.441369 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:18.441312    2563 scope.go:117] "RemoveContainer" containerID="e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b"
Sep 10 12:56:18.441654 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:56:18.441632    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-vjw4t_kuadrant-system(ce40549a-16c6-4438-995e-01f1f68546da)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t" podUID="ce40549a-16c6-4438-995e-01f1f68546da"
Sep 10 12:56:20.660161 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:20.660118    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:56:20.660710 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:20.660662    2563 scope.go:117] "RemoveContainer" containerID="e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b"
Sep 10 12:56:20.660975 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:56:20.660948    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-vjw4t_kuadrant-system(ce40549a-16c6-4438-995e-01f1f68546da)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t" podUID="ce40549a-16c6-4438-995e-01f1f68546da"
Sep 10 12:56:27.558109 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:27.558062    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:56:27.558617 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:27.558600    2563 scope.go:117] "RemoveContainer" containerID="e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b"
Sep 10 12:56:27.558867 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:56:27.558842    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=kuadrant-operator-controller-manager-5769f99d9f-vjw4t_kuadrant-system(ce40549a-16c6-4438-995e-01f1f68546da)\"" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t" podUID="ce40549a-16c6-4438-995e-01f1f68546da"
Sep 10 12:56:39.698996 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:39.698960    2563 scope.go:117] "RemoveContainer" containerID="e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b"
Sep 10 12:56:40.596914 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:40.596873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t" event={"ID":"ce40549a-16c6-4438-995e-01f1f68546da","Type":"ContainerStarted","Data":"266d5f51ff1f2ccdf069998780dfa4db299f603502074c20558672f1da1b6f6d"}
Sep 10 12:56:40.597150 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:40.597132    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:56:46.084039 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.084001    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"]
Sep 10 12:56:46.084592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.084563    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50bde372-ac63-42be-b27c-c87a7a3f8d6d" containerName="maas-api"
Sep 10 12:56:46.084592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.084586    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="50bde372-ac63-42be-b27c-c87a7a3f8d6d" containerName="maas-api"
Sep 10 12:56:46.084755 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.084683    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="50bde372-ac63-42be-b27c-c87a7a3f8d6d" containerName="maas-api"
Sep 10 12:56:46.088282 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.088258    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.093945 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.093916    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-fe8579-serving-cert\""
Sep 10 12:56:46.161815 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.161775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a30fdeb-971c-41de-8adc-1d047fd15c9d-metrics-certs\") pod \"maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.162048 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.161881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a30fdeb-971c-41de-8adc-1d047fd15c9d-maas-api-tls\") pod \"maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.162048 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.161915    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a30fdeb-971c-41de-8adc-1d047fd15c9d-openshift-service-ca\") pod \"maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.162048 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.161978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szjq\" (UniqueName: \"kubernetes.io/projected/6a30fdeb-971c-41de-8adc-1d047fd15c9d-kube-api-access-7szjq\") pod \"maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.169048 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.169008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"]
Sep 10 12:56:46.263009 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.262960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a30fdeb-971c-41de-8adc-1d047fd15c9d-openshift-service-ca\") pod \"maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.263198 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.263027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szjq\" (UniqueName: \"kubernetes.io/projected/6a30fdeb-971c-41de-8adc-1d047fd15c9d-kube-api-access-7szjq\") pod \"maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.263198 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.263182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a30fdeb-971c-41de-8adc-1d047fd15c9d-metrics-certs\") pod \"maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.263282 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.263265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a30fdeb-971c-41de-8adc-1d047fd15c9d-maas-api-tls\") pod \"maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.263719 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.263694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a30fdeb-971c-41de-8adc-1d047fd15c9d-openshift-service-ca\") pod \"maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.265819 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.265796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a30fdeb-971c-41de-8adc-1d047fd15c9d-metrics-certs\") pod \"maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.266134 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.266097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a30fdeb-971c-41de-8adc-1d047fd15c9d-maas-api-tls\") pod \"maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.279894 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.279865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szjq\" (UniqueName: \"kubernetes.io/projected/6a30fdeb-971c-41de-8adc-1d047fd15c9d-kube-api-access-7szjq\") pod \"maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.399349 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.399251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:46.692635 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.692603    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"]
Sep 10 12:56:46.693692 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:56:46.693665    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a30fdeb_971c_41de_8adc_1d047fd15c9d.slice/crio-2c35786a9e1e1b8e61649740e1040bf4914cca94878b3ffc6973774051531085 WatchSource:0}: Error finding container 2c35786a9e1e1b8e61649740e1040bf4914cca94878b3ffc6973774051531085: Status 404 returned error can't find the container with id 2c35786a9e1e1b8e61649740e1040bf4914cca94878b3ffc6973774051531085
Sep 10 12:56:46.695064 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:46.695045    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:56:47.341502 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.341468    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"]
Sep 10 12:56:47.347388 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.347346    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"
Sep 10 12:56:47.353866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.351793    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-fe8579\""
Sep 10 12:56:47.353866 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.352099    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-fe8579-dockercfg-zzpbn\""
Sep 10 12:56:47.367671 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.367643    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"]
Sep 10 12:56:47.477216 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.477180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66cc365-c450-4c4e-990a-3b4bd233320c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk\" (UID: \"f66cc365-c450-4c4e-990a-3b4bd233320c\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"
Sep 10 12:56:47.477391 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.477285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjrw\" (UniqueName: \"kubernetes.io/projected/f66cc365-c450-4c4e-990a-3b4bd233320c-kube-api-access-jzjrw\") pod \"payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk\" (UID: \"f66cc365-c450-4c4e-990a-3b4bd233320c\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"
Sep 10 12:56:47.578644 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.578520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjrw\" (UniqueName: \"kubernetes.io/projected/f66cc365-c450-4c4e-990a-3b4bd233320c-kube-api-access-jzjrw\") pod \"payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk\" (UID: \"f66cc365-c450-4c4e-990a-3b4bd233320c\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"
Sep 10 12:56:47.578820 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.578681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66cc365-c450-4c4e-990a-3b4bd233320c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk\" (UID: \"f66cc365-c450-4c4e-990a-3b4bd233320c\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"
Sep 10 12:56:47.579420 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.579398    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66cc365-c450-4c4e-990a-3b4bd233320c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk\" (UID: \"f66cc365-c450-4c4e-990a-3b4bd233320c\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"
Sep 10 12:56:47.604031 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.603994    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjrw\" (UniqueName: \"kubernetes.io/projected/f66cc365-c450-4c4e-990a-3b4bd233320c-kube-api-access-jzjrw\") pod \"payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk\" (UID: \"f66cc365-c450-4c4e-990a-3b4bd233320c\") " pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"
Sep 10 12:56:47.644592 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.644533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42" event={"ID":"6a30fdeb-971c-41de-8adc-1d047fd15c9d","Type":"ContainerStarted","Data":"55f831777e48b581ca74c4dbf597c321e5f15f2d51d0f165be83509d70a31f7d"}
Sep 10 12:56:47.644762 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.644631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42" event={"ID":"6a30fdeb-971c-41de-8adc-1d047fd15c9d","Type":"ContainerStarted","Data":"2c35786a9e1e1b8e61649740e1040bf4914cca94878b3ffc6973774051531085"}
Sep 10 12:56:47.644762 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.644654    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:47.672425 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.672367    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42" podStartSLOduration=1.044628593 podStartE2EDuration="1.672350379s" totalImagesPullingTime="627.721786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:56:47.670336555 +0000 UTC m=+1581.595248043" watchObservedRunningTime="2026-09-10 12:56:47.672350379 +0000 UTC m=+1581.597261868"
Sep 10 12:56:47.691484 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.691452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"
Sep 10 12:56:47.994492 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:47.994466    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"]
Sep 10 12:56:47.996287 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:56:47.996256    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66cc365_c450_4c4e_990a_3b4bd233320c.slice/crio-2898e183bdabab5b934e592f3b55bbaed1a45c21c248f11b44c308643a57adf0 WatchSource:0}: Error finding container 2898e183bdabab5b934e592f3b55bbaed1a45c21c248f11b44c308643a57adf0: Status 404 returned error can't find the container with id 2898e183bdabab5b934e592f3b55bbaed1a45c21c248f11b44c308643a57adf0
Sep 10 12:56:48.654386 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:48.654341    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk" event={"ID":"f66cc365-c450-4c4e-990a-3b4bd233320c","Type":"ContainerStarted","Data":"4822111003a44efdbec0c3e1034f11ecbeb8e6d7d1591f127f09a1d8358105b3"}
Sep 10 12:56:48.654970 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:48.654416    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"
Sep 10 12:56:48.654970 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:48.654440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk" event={"ID":"f66cc365-c450-4c4e-990a-3b4bd233320c","Type":"ContainerStarted","Data":"2898e183bdabab5b934e592f3b55bbaed1a45c21c248f11b44c308643a57adf0"}
Sep 10 12:56:48.775434 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:48.775376    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk" podStartSLOduration=1.7753611550000001 podStartE2EDuration="1.775361155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:56:48.775128141 +0000 UTC m=+1582.700039628" watchObservedRunningTime="2026-09-10 12:56:48.775361155 +0000 UTC m=+1582.700272643"
Sep 10 12:56:51.605137 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:51.605102    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 12:56:53.661139 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:53.661100    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:56:59.663784 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:56:59.663745    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"
Sep 10 12:57:09.430278 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:09.430233    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:57:15.040722 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.040571    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"]
Sep 10 12:57:15.052133 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.052059    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.056218 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.056183    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b3c07b-serving-cert\""
Sep 10 12:57:15.067509 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.059286    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"]
Sep 10 12:57:15.186131 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.186088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-metrics-certs\") pod \"maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.188028 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.186160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.188028 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.186234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2fcs\" (UniqueName: \"kubernetes.io/projected/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-kube-api-access-t2fcs\") pod \"maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.188028 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.186366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-maas-api-tls\") pod \"maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.288104 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.288059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.288326 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.288132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fcs\" (UniqueName: \"kubernetes.io/projected/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-kube-api-access-t2fcs\") pod \"maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.288326 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.288234    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-maas-api-tls\") pod \"maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.288326 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.288272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-metrics-certs\") pod \"maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.288831 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.288806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-openshift-service-ca\") pod \"maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.290868 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.290821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-metrics-certs\") pod \"maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.290868 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.290851    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-maas-api-tls\") pod \"maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.311382 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.311343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2fcs\" (UniqueName: \"kubernetes.io/projected/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-kube-api-access-t2fcs\") pod \"maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.381396 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.381349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:15.680694 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.680663    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"]
Sep 10 12:57:15.683762 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:57:15.683720    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996b2c57_a1d0_4e6a_8f09_8da1c16c4991.slice/crio-f5712c37b6d4477d70a3e3404b66eb0f21def46c9fc39b1aa88cd8099b4be66f WatchSource:0}: Error finding container f5712c37b6d4477d70a3e3404b66eb0f21def46c9fc39b1aa88cd8099b4be66f: Status 404 returned error can't find the container with id f5712c37b6d4477d70a3e3404b66eb0f21def46c9fc39b1aa88cd8099b4be66f
Sep 10 12:57:15.831451 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:15.831415    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42" event={"ID":"996b2c57-a1d0-4e6a-8f09-8da1c16c4991","Type":"ContainerStarted","Data":"f5712c37b6d4477d70a3e3404b66eb0f21def46c9fc39b1aa88cd8099b4be66f"}
Sep 10 12:57:16.839872 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:16.839830    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42" event={"ID":"996b2c57-a1d0-4e6a-8f09-8da1c16c4991","Type":"ContainerStarted","Data":"e6fe26726eda4a4ba421528fe1033a07c12ba22b81845373e9bfdd1126c13a9a"}
Sep 10 12:57:16.840364 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:16.839892    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:16.884940 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:16.884870    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42" podStartSLOduration=1.56347532 podStartE2EDuration="1.884851425s" totalImagesPullingTime="321.376105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:57:16.884708465 +0000 UTC m=+1610.809619953" watchObservedRunningTime="2026-09-10 12:57:16.884851425 +0000 UTC m=+1610.809762912"
Sep 10 12:57:22.853255 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:22.853164    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:57:26.834404 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:26.834364    2563 scope.go:117] "RemoveContainer" containerID="1af696f8916b53e69138be40d3f3c83e9d3ac6d41f1a9f572e15f2db16245e18"
Sep 10 12:57:57.542572 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.542505    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"]
Sep 10 12:57:57.543141 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.542768    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42" podUID="6a30fdeb-971c-41de-8adc-1d047fd15c9d" containerName="maas-api" containerID="cri-o://55f831777e48b581ca74c4dbf597c321e5f15f2d51d0f165be83509d70a31f7d" gracePeriod=30
Sep 10 12:57:57.642819 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.642782    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"]
Sep 10 12:57:57.643053 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.643028    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk" podUID="f66cc365-c450-4c4e-990a-3b4bd233320c" containerName="payload-processing" containerID="cri-o://4822111003a44efdbec0c3e1034f11ecbeb8e6d7d1591f127f09a1d8358105b3" gracePeriod=30
Sep 10 12:57:57.821126 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.821095    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:57:57.926893 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.926863    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"
Sep 10 12:57:57.971729 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.971689    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a30fdeb-971c-41de-8adc-1d047fd15c9d-openshift-service-ca\") pod \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") "
Sep 10 12:57:57.971922 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.971865    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a30fdeb-971c-41de-8adc-1d047fd15c9d-metrics-certs\") pod \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") "
Sep 10 12:57:57.971973 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.971937    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a30fdeb-971c-41de-8adc-1d047fd15c9d-maas-api-tls\") pod \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") "
Sep 10 12:57:57.972081 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.972059    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7szjq\" (UniqueName: \"kubernetes.io/projected/6a30fdeb-971c-41de-8adc-1d047fd15c9d-kube-api-access-7szjq\") pod \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\" (UID: \"6a30fdeb-971c-41de-8adc-1d047fd15c9d\") "
Sep 10 12:57:57.972140 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.972113    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a30fdeb-971c-41de-8adc-1d047fd15c9d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a30fdeb-971c-41de-8adc-1d047fd15c9d" (UID: "6a30fdeb-971c-41de-8adc-1d047fd15c9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:57:57.972626 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.972598    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a30fdeb-971c-41de-8adc-1d047fd15c9d-openshift-service-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:57:57.974110 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.974084    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a30fdeb-971c-41de-8adc-1d047fd15c9d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a30fdeb-971c-41de-8adc-1d047fd15c9d" (UID: "6a30fdeb-971c-41de-8adc-1d047fd15c9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:57:57.974227 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.974130    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a30fdeb-971c-41de-8adc-1d047fd15c9d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6a30fdeb-971c-41de-8adc-1d047fd15c9d" (UID: "6a30fdeb-971c-41de-8adc-1d047fd15c9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:57:57.974227 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:57.974181    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a30fdeb-971c-41de-8adc-1d047fd15c9d-kube-api-access-7szjq" (OuterVolumeSpecName: "kube-api-access-7szjq") pod "6a30fdeb-971c-41de-8adc-1d047fd15c9d" (UID: "6a30fdeb-971c-41de-8adc-1d047fd15c9d"). InnerVolumeSpecName "kube-api-access-7szjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:57:58.073951 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.073849    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzjrw\" (UniqueName: \"kubernetes.io/projected/f66cc365-c450-4c4e-990a-3b4bd233320c-kube-api-access-jzjrw\") pod \"f66cc365-c450-4c4e-990a-3b4bd233320c\" (UID: \"f66cc365-c450-4c4e-990a-3b4bd233320c\") "
Sep 10 12:57:58.074117 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.073952    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66cc365-c450-4c4e-990a-3b4bd233320c-plugins-config-volume\") pod \"f66cc365-c450-4c4e-990a-3b4bd233320c\" (UID: \"f66cc365-c450-4c4e-990a-3b4bd233320c\") "
Sep 10 12:57:58.074310 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.074285    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66cc365-c450-4c4e-990a-3b4bd233320c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f66cc365-c450-4c4e-990a-3b4bd233320c" (UID: "f66cc365-c450-4c4e-990a-3b4bd233320c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:57:58.074447 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.074428    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a30fdeb-971c-41de-8adc-1d047fd15c9d-metrics-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:57:58.074485 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.074451    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f66cc365-c450-4c4e-990a-3b4bd233320c-plugins-config-volume\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:57:58.074485 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.074462    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a30fdeb-971c-41de-8adc-1d047fd15c9d-maas-api-tls\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:57:58.074485 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.074475    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7szjq\" (UniqueName: \"kubernetes.io/projected/6a30fdeb-971c-41de-8adc-1d047fd15c9d-kube-api-access-7szjq\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:57:58.076080 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.076057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66cc365-c450-4c4e-990a-3b4bd233320c-kube-api-access-jzjrw" (OuterVolumeSpecName: "kube-api-access-jzjrw") pod "f66cc365-c450-4c4e-990a-3b4bd233320c" (UID: "f66cc365-c450-4c4e-990a-3b4bd233320c"). InnerVolumeSpecName "kube-api-access-jzjrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:57:58.104160 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.104123    2563 generic.go:356] "Generic (PLEG): container finished" podID="6a30fdeb-971c-41de-8adc-1d047fd15c9d" containerID="55f831777e48b581ca74c4dbf597c321e5f15f2d51d0f165be83509d70a31f7d" exitCode=0
Sep 10 12:57:58.104334 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.104172    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"
Sep 10 12:57:58.104334 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.104210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42" event={"ID":"6a30fdeb-971c-41de-8adc-1d047fd15c9d","Type":"ContainerDied","Data":"55f831777e48b581ca74c4dbf597c321e5f15f2d51d0f165be83509d70a31f7d"}
Sep 10 12:57:58.104334 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.104260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42" event={"ID":"6a30fdeb-971c-41de-8adc-1d047fd15c9d","Type":"ContainerDied","Data":"2c35786a9e1e1b8e61649740e1040bf4914cca94878b3ffc6973774051531085"}
Sep 10 12:57:58.104334 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.104288    2563 scope.go:117] "RemoveContainer" containerID="55f831777e48b581ca74c4dbf597c321e5f15f2d51d0f165be83509d70a31f7d"
Sep 10 12:57:58.105639 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.105612    2563 generic.go:356] "Generic (PLEG): container finished" podID="f66cc365-c450-4c4e-990a-3b4bd233320c" containerID="4822111003a44efdbec0c3e1034f11ecbeb8e6d7d1591f127f09a1d8358105b3" exitCode=0
Sep 10 12:57:58.105730 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.105673    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"
Sep 10 12:57:58.105730 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.105691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk" event={"ID":"f66cc365-c450-4c4e-990a-3b4bd233320c","Type":"ContainerDied","Data":"4822111003a44efdbec0c3e1034f11ecbeb8e6d7d1591f127f09a1d8358105b3"}
Sep 10 12:57:58.105853 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.105730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk" event={"ID":"f66cc365-c450-4c4e-990a-3b4bd233320c","Type":"ContainerDied","Data":"2898e183bdabab5b934e592f3b55bbaed1a45c21c248f11b44c308643a57adf0"}
Sep 10 12:57:58.119054 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.119022    2563 scope.go:117] "RemoveContainer" containerID="55f831777e48b581ca74c4dbf597c321e5f15f2d51d0f165be83509d70a31f7d"
Sep 10 12:57:58.119600 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:57:58.119526    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55f831777e48b581ca74c4dbf597c321e5f15f2d51d0f165be83509d70a31f7d\": container with ID starting with 55f831777e48b581ca74c4dbf597c321e5f15f2d51d0f165be83509d70a31f7d not found: ID does not exist" containerID="55f831777e48b581ca74c4dbf597c321e5f15f2d51d0f165be83509d70a31f7d"
Sep 10 12:57:58.119716 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.119611    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55f831777e48b581ca74c4dbf597c321e5f15f2d51d0f165be83509d70a31f7d"} err="failed to get container status \"55f831777e48b581ca74c4dbf597c321e5f15f2d51d0f165be83509d70a31f7d\": rpc error: code = NotFound desc = could not find container \"55f831777e48b581ca74c4dbf597c321e5f15f2d51d0f165be83509d70a31f7d\": container with ID starting with 55f831777e48b581ca74c4dbf597c321e5f15f2d51d0f165be83509d70a31f7d not found: ID does not exist"
Sep 10 12:57:58.119716 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.119637    2563 scope.go:117] "RemoveContainer" containerID="4822111003a44efdbec0c3e1034f11ecbeb8e6d7d1591f127f09a1d8358105b3"
Sep 10 12:57:58.132512 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.132481    2563 scope.go:117] "RemoveContainer" containerID="4822111003a44efdbec0c3e1034f11ecbeb8e6d7d1591f127f09a1d8358105b3"
Sep 10 12:57:58.132992 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:57:58.132960    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4822111003a44efdbec0c3e1034f11ecbeb8e6d7d1591f127f09a1d8358105b3\": container with ID starting with 4822111003a44efdbec0c3e1034f11ecbeb8e6d7d1591f127f09a1d8358105b3 not found: ID does not exist" containerID="4822111003a44efdbec0c3e1034f11ecbeb8e6d7d1591f127f09a1d8358105b3"
Sep 10 12:57:58.133117 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.133003    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4822111003a44efdbec0c3e1034f11ecbeb8e6d7d1591f127f09a1d8358105b3"} err="failed to get container status \"4822111003a44efdbec0c3e1034f11ecbeb8e6d7d1591f127f09a1d8358105b3\": rpc error: code = NotFound desc = could not find container \"4822111003a44efdbec0c3e1034f11ecbeb8e6d7d1591f127f09a1d8358105b3\": container with ID starting with 4822111003a44efdbec0c3e1034f11ecbeb8e6d7d1591f127f09a1d8358105b3 not found: ID does not exist"
Sep 10 12:57:58.135870 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.135843    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"]
Sep 10 12:57:58.138426 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.138402    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-fe8579-58b95ccd67-l8s42"]
Sep 10 12:57:58.154691 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.154649    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"]
Sep 10 12:57:58.160724 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.160691    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-fe8579-85cdd6d7c9-47mkk"]
Sep 10 12:57:58.176085 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.176053    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzjrw\" (UniqueName: \"kubernetes.io/projected/f66cc365-c450-4c4e-990a-3b4bd233320c-kube-api-access-jzjrw\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:57:58.706345 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.706300    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a30fdeb-971c-41de-8adc-1d047fd15c9d" path="/var/lib/kubelet/pods/6a30fdeb-971c-41de-8adc-1d047fd15c9d/volumes"
Sep 10 12:57:58.706785 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.706740    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f66cc365-c450-4c4e-990a-3b4bd233320c" path="/var/lib/kubelet/pods/f66cc365-c450-4c4e-990a-3b4bd233320c/volumes"
Sep 10 12:57:58.827002 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:57:58.826904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:58:21.996910 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:21.996871    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"]
Sep 10 12:58:21.997543 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:21.997125    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42" podUID="996b2c57-a1d0-4e6a-8f09-8da1c16c4991" containerName="maas-api" containerID="cri-o://e6fe26726eda4a4ba421528fe1033a07c12ba22b81845373e9bfdd1126c13a9a" gracePeriod=30
Sep 10 12:58:22.260797 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.260716    2563 generic.go:356] "Generic (PLEG): container finished" podID="996b2c57-a1d0-4e6a-8f09-8da1c16c4991" containerID="e6fe26726eda4a4ba421528fe1033a07c12ba22b81845373e9bfdd1126c13a9a" exitCode=0
Sep 10 12:58:22.260950 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.260792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42" event={"ID":"996b2c57-a1d0-4e6a-8f09-8da1c16c4991","Type":"ContainerDied","Data":"e6fe26726eda4a4ba421528fe1033a07c12ba22b81845373e9bfdd1126c13a9a"}
Sep 10 12:58:22.260950 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.260842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42" event={"ID":"996b2c57-a1d0-4e6a-8f09-8da1c16c4991","Type":"ContainerDied","Data":"f5712c37b6d4477d70a3e3404b66eb0f21def46c9fc39b1aa88cd8099b4be66f"}
Sep 10 12:58:22.260950 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.260859    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5712c37b6d4477d70a3e3404b66eb0f21def46c9fc39b1aa88cd8099b4be66f"
Sep 10 12:58:22.264595 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.264568    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:58:22.353502 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.353460    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-metrics-certs\") pod \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") "
Sep 10 12:58:22.353502 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.353502    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-openshift-service-ca\") pod \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") "
Sep 10 12:58:22.353793 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.353526    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2fcs\" (UniqueName: \"kubernetes.io/projected/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-kube-api-access-t2fcs\") pod \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") "
Sep 10 12:58:22.353793 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.353605    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-maas-api-tls\") pod \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\" (UID: \"996b2c57-a1d0-4e6a-8f09-8da1c16c4991\") "
Sep 10 12:58:22.353903 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.353873    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "996b2c57-a1d0-4e6a-8f09-8da1c16c4991" (UID: "996b2c57-a1d0-4e6a-8f09-8da1c16c4991"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:58:22.354095 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.354072    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-openshift-service-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:22.356249 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.356217    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "996b2c57-a1d0-4e6a-8f09-8da1c16c4991" (UID: "996b2c57-a1d0-4e6a-8f09-8da1c16c4991"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:58:22.356377 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.356236    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "996b2c57-a1d0-4e6a-8f09-8da1c16c4991" (UID: "996b2c57-a1d0-4e6a-8f09-8da1c16c4991"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:58:22.356377 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.356249    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-kube-api-access-t2fcs" (OuterVolumeSpecName: "kube-api-access-t2fcs") pod "996b2c57-a1d0-4e6a-8f09-8da1c16c4991" (UID: "996b2c57-a1d0-4e6a-8f09-8da1c16c4991"). InnerVolumeSpecName "kube-api-access-t2fcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:58:22.454780 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.454744    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-maas-api-tls\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:22.454780 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.454776    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-metrics-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:22.454780 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:22.454786    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2fcs\" (UniqueName: \"kubernetes.io/projected/996b2c57-a1d0-4e6a-8f09-8da1c16c4991-kube-api-access-t2fcs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:23.267893 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:23.267858    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"
Sep 10 12:58:23.290935 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:23.290889    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"]
Sep 10 12:58:23.295891 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:23.295844    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b3c07b-77787c5bbc-2gz42"]
Sep 10 12:58:24.713348 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:24.713302    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="996b2c57-a1d0-4e6a-8f09-8da1c16c4991" path="/var/lib/kubelet/pods/996b2c57-a1d0-4e6a-8f09-8da1c16c4991/volumes"
Sep 10 12:58:30.351961 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.351915    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7"]
Sep 10 12:58:30.352485 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.352469    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="996b2c57-a1d0-4e6a-8f09-8da1c16c4991" containerName="maas-api"
Sep 10 12:58:30.352485 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.352486    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="996b2c57-a1d0-4e6a-8f09-8da1c16c4991" containerName="maas-api"
Sep 10 12:58:30.352603 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.352506    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a30fdeb-971c-41de-8adc-1d047fd15c9d" containerName="maas-api"
Sep 10 12:58:30.352603 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.352515    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a30fdeb-971c-41de-8adc-1d047fd15c9d" containerName="maas-api"
Sep 10 12:58:30.352603 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.352529    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f66cc365-c450-4c4e-990a-3b4bd233320c" containerName="payload-processing"
Sep 10 12:58:30.352603 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.352536    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f66cc365-c450-4c4e-990a-3b4bd233320c" containerName="payload-processing"
Sep 10 12:58:30.352736 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.352658    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f66cc365-c450-4c4e-990a-3b4bd233320c" containerName="payload-processing"
Sep 10 12:58:30.352736 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.352670    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a30fdeb-971c-41de-8adc-1d047fd15c9d" containerName="maas-api"
Sep 10 12:58:30.352736 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.352682    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="996b2c57-a1d0-4e6a-8f09-8da1c16c4991" containerName="maas-api"
Sep 10 12:58:30.355885 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.355865    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7"
Sep 10 12:58:30.359335 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.359310    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 12:58:30.366855 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.366821    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7"]
Sep 10 12:58:30.438840 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.438788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27637cc5-7e74-43fe-9300-b51fcfe9a40d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7\" (UID: \"27637cc5-7e74-43fe-9300-b51fcfe9a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7"
Sep 10 12:58:30.540537 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.540491    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27637cc5-7e74-43fe-9300-b51fcfe9a40d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7\" (UID: \"27637cc5-7e74-43fe-9300-b51fcfe9a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7"
Sep 10 12:58:30.541236 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.541215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27637cc5-7e74-43fe-9300-b51fcfe9a40d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7\" (UID: \"27637cc5-7e74-43fe-9300-b51fcfe9a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7"
Sep 10 12:58:30.668541 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.668508    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7"
Sep 10 12:58:30.977725 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:30.977691    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7"]
Sep 10 12:58:30.978718 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:58:30.978689    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27637cc5_7e74_43fe_9300_b51fcfe9a40d.slice/crio-53ed12a1932d12adee3b856873890bffda7982b7bc7153ecdec7f052b3fac909 WatchSource:0}: Error finding container 53ed12a1932d12adee3b856873890bffda7982b7bc7153ecdec7f052b3fac909: Status 404 returned error can't find the container with id 53ed12a1932d12adee3b856873890bffda7982b7bc7153ecdec7f052b3fac909
Sep 10 12:58:31.321167 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:31.321126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7" event={"ID":"27637cc5-7e74-43fe-9300-b51fcfe9a40d","Type":"ContainerStarted","Data":"c1e8e51d082246dfd7fce283464c74f48647614bed57af9acb83290be52819de"}
Sep 10 12:58:31.321370 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:31.321190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7" event={"ID":"27637cc5-7e74-43fe-9300-b51fcfe9a40d","Type":"ContainerStarted","Data":"53ed12a1932d12adee3b856873890bffda7982b7bc7153ecdec7f052b3fac909"}
Sep 10 12:58:31.343894 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:31.343827    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7" podStartSLOduration=1.343805675 podStartE2EDuration="1.343805675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:58:31.341100338 +0000 UTC m=+1685.266011825" watchObservedRunningTime="2026-09-10 12:58:31.343805675 +0000 UTC m=+1685.268717162"
Sep 10 12:58:32.330464 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:32.330422    2563 generic.go:356] "Generic (PLEG): container finished" podID="27637cc5-7e74-43fe-9300-b51fcfe9a40d" containerID="c1e8e51d082246dfd7fce283464c74f48647614bed57af9acb83290be52819de" exitCode=0
Sep 10 12:58:32.330936 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:32.330462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7" event={"ID":"27637cc5-7e74-43fe-9300-b51fcfe9a40d","Type":"ContainerDied","Data":"c1e8e51d082246dfd7fce283464c74f48647614bed57af9acb83290be52819de"}
Sep 10 12:58:33.485103 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:33.485073    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7"
Sep 10 12:58:33.573501 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:33.573462    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27637cc5-7e74-43fe-9300-b51fcfe9a40d-maas-api-service-ca\") pod \"27637cc5-7e74-43fe-9300-b51fcfe9a40d\" (UID: \"27637cc5-7e74-43fe-9300-b51fcfe9a40d\") "
Sep 10 12:58:33.573873 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:33.573845    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27637cc5-7e74-43fe-9300-b51fcfe9a40d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "27637cc5-7e74-43fe-9300-b51fcfe9a40d" (UID: "27637cc5-7e74-43fe-9300-b51fcfe9a40d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:58:33.675278 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:33.675230    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27637cc5-7e74-43fe-9300-b51fcfe9a40d-maas-api-service-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:34.344717 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:34.344681    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7"
Sep 10 12:58:34.344901 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:34.344717    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7" event={"ID":"27637cc5-7e74-43fe-9300-b51fcfe9a40d","Type":"ContainerDied","Data":"53ed12a1932d12adee3b856873890bffda7982b7bc7153ecdec7f052b3fac909"}
Sep 10 12:58:34.344901 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:34.344767    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53ed12a1932d12adee3b856873890bffda7982b7bc7153ecdec7f052b3fac909"
Sep 10 12:58:35.103040 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:35.102993    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"]
Sep 10 12:58:35.112758 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:35.112725    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"]
Sep 10 12:58:35.137172 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:35.137138    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"]
Sep 10 12:58:35.151604 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:35.151567    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"]
Sep 10 12:58:45.326373 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:45.326330    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:58:46.618342 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:46.618297    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7"]
Sep 10 12:58:46.620585 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:46.620560    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-1312ea-tjvc7"]
Sep 10 12:58:46.708074 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:46.708030    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27637cc5-7e74-43fe-9300-b51fcfe9a40d" path="/var/lib/kubelet/pods/27637cc5-7e74-43fe-9300-b51fcfe9a40d/volumes"
Sep 10 12:58:50.124650 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.124539    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579" podUID="b03a628d-daed-477f-aa0d-bdfbbb50ef7e" containerName="main" containerID="cri-o://a7801efcff397bfb14b6deff56364afa3fa593ad40a4e4c432a91bf986efb4ce" gracePeriod=45
Sep 10 12:58:50.135147 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.135103    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc" podUID="6f6d5312-fdb6-40e6-80bb-703f052a0910" containerName="main" containerID="cri-o://1af326bc72279cb50d9312f5fa472ec6baaa299e92ca6eec595b5e0d2c1bf41c" gracePeriod=45
Sep 10 12:58:50.160653 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.160607    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm" podUID="d941e342-015d-45b4-86eb-d0f2831388bf" containerName="main" containerID="cri-o://e05ab99686a752d2c251e5441992996efa356f6e108618048ec21fe5d848b35e" gracePeriod=45
Sep 10 12:58:50.173685 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.173639    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk" podUID="144c7178-a13b-4068-8866-5c0592344252" containerName="main" containerID="cri-o://f31160967042beea125449a8a5c4513456fb7986765d92fa8b830e807c2b8601" gracePeriod=45
Sep 10 12:58:50.457494 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.456240    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:58:50.460937 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.460860    2563 generic.go:356] "Generic (PLEG): container finished" podID="144c7178-a13b-4068-8866-5c0592344252" containerID="f31160967042beea125449a8a5c4513456fb7986765d92fa8b830e807c2b8601" exitCode=0
Sep 10 12:58:50.461129 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.460955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk" event={"ID":"144c7178-a13b-4068-8866-5c0592344252","Type":"ContainerDied","Data":"f31160967042beea125449a8a5c4513456fb7986765d92fa8b830e807c2b8601"}
Sep 10 12:58:50.463273 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.463245    2563 generic.go:356] "Generic (PLEG): container finished" podID="b03a628d-daed-477f-aa0d-bdfbbb50ef7e" containerID="a7801efcff397bfb14b6deff56364afa3fa593ad40a4e4c432a91bf986efb4ce" exitCode=0
Sep 10 12:58:50.463424 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.463293    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579" event={"ID":"b03a628d-daed-477f-aa0d-bdfbbb50ef7e","Type":"ContainerDied","Data":"a7801efcff397bfb14b6deff56364afa3fa593ad40a4e4c432a91bf986efb4ce"}
Sep 10 12:58:50.465634 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.465544    2563 generic.go:356] "Generic (PLEG): container finished" podID="d941e342-015d-45b4-86eb-d0f2831388bf" containerID="e05ab99686a752d2c251e5441992996efa356f6e108618048ec21fe5d848b35e" exitCode=0
Sep 10 12:58:50.465784 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.465666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm" event={"ID":"d941e342-015d-45b4-86eb-d0f2831388bf","Type":"ContainerDied","Data":"e05ab99686a752d2c251e5441992996efa356f6e108618048ec21fe5d848b35e"}
Sep 10 12:58:50.467960 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.467925    2563 generic.go:356] "Generic (PLEG): container finished" podID="6f6d5312-fdb6-40e6-80bb-703f052a0910" containerID="1af326bc72279cb50d9312f5fa472ec6baaa299e92ca6eec595b5e0d2c1bf41c" exitCode=0
Sep 10 12:58:50.468098 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.467994    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc" event={"ID":"6f6d5312-fdb6-40e6-80bb-703f052a0910","Type":"ContainerDied","Data":"1af326bc72279cb50d9312f5fa472ec6baaa299e92ca6eec595b5e0d2c1bf41c"}
Sep 10 12:58:50.468098 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.468030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc" event={"ID":"6f6d5312-fdb6-40e6-80bb-703f052a0910","Type":"ContainerDied","Data":"76f51cb648d3067afbc14045aa7b01de5e9731c795df49b4a3ee4f91c8655e76"}
Sep 10 12:58:50.468098 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.468052    2563 scope.go:117] "RemoveContainer" containerID="1af326bc72279cb50d9312f5fa472ec6baaa299e92ca6eec595b5e0d2c1bf41c"
Sep 10 12:58:50.468266 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.468156    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"
Sep 10 12:58:50.500969 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.499241    2563 scope.go:117] "RemoveContainer" containerID="1af326bc72279cb50d9312f5fa472ec6baaa299e92ca6eec595b5e0d2c1bf41c"
Sep 10 12:58:50.502631 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:58:50.502590    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1af326bc72279cb50d9312f5fa472ec6baaa299e92ca6eec595b5e0d2c1bf41c\": container with ID starting with 1af326bc72279cb50d9312f5fa472ec6baaa299e92ca6eec595b5e0d2c1bf41c not found: ID does not exist" containerID="1af326bc72279cb50d9312f5fa472ec6baaa299e92ca6eec595b5e0d2c1bf41c"
Sep 10 12:58:50.502798 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.502644    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1af326bc72279cb50d9312f5fa472ec6baaa299e92ca6eec595b5e0d2c1bf41c"} err="failed to get container status \"1af326bc72279cb50d9312f5fa472ec6baaa299e92ca6eec595b5e0d2c1bf41c\": rpc error: code = NotFound desc = could not find container \"1af326bc72279cb50d9312f5fa472ec6baaa299e92ca6eec595b5e0d2c1bf41c\": container with ID starting with 1af326bc72279cb50d9312f5fa472ec6baaa299e92ca6eec595b5e0d2c1bf41c not found: ID does not exist"
Sep 10 12:58:50.532796 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.532760    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:58:50.553225 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.553197    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:58:50.560605 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.560574    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m8xq\" (UniqueName: \"kubernetes.io/projected/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-kube-api-access-5m8xq\") pod \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") "
Sep 10 12:58:50.560768 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.560713    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-dshm\") pod \"6f6d5312-fdb6-40e6-80bb-703f052a0910\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") "
Sep 10 12:58:50.560768 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.560745    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-dshm\") pod \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") "
Sep 10 12:58:50.560959 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.560789    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcpmq\" (UniqueName: \"kubernetes.io/projected/6f6d5312-fdb6-40e6-80bb-703f052a0910-kube-api-access-lcpmq\") pod \"6f6d5312-fdb6-40e6-80bb-703f052a0910\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") "
Sep 10 12:58:50.560959 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.560820    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-home\") pod \"6f6d5312-fdb6-40e6-80bb-703f052a0910\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") "
Sep 10 12:58:50.560959 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.560870    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-tmp-dir\") pod \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") "
Sep 10 12:58:50.560959 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.560912    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-home\") pod \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") "
Sep 10 12:58:50.561163 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.560962    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f6d5312-fdb6-40e6-80bb-703f052a0910-tls-certs\") pod \"6f6d5312-fdb6-40e6-80bb-703f052a0910\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") "
Sep 10 12:58:50.561163 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.560997    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-model-cache\") pod \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") "
Sep 10 12:58:50.561163 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.561038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-model-cache\") pod \"6f6d5312-fdb6-40e6-80bb-703f052a0910\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") "
Sep 10 12:58:50.561163 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.561060    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-tmp-dir\") pod \"6f6d5312-fdb6-40e6-80bb-703f052a0910\" (UID: \"6f6d5312-fdb6-40e6-80bb-703f052a0910\") "
Sep 10 12:58:50.561163 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.561110    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-tls-certs\") pod \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\" (UID: \"b03a628d-daed-477f-aa0d-bdfbbb50ef7e\") "
Sep 10 12:58:50.561483 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.561342    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-home" (OuterVolumeSpecName: "home") pod "b03a628d-daed-477f-aa0d-bdfbbb50ef7e" (UID: "b03a628d-daed-477f-aa0d-bdfbbb50ef7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.562150 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.561659    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.562150 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.561881    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-home" (OuterVolumeSpecName: "home") pod "6f6d5312-fdb6-40e6-80bb-703f052a0910" (UID: "6f6d5312-fdb6-40e6-80bb-703f052a0910"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.562150 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.561926    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b03a628d-daed-477f-aa0d-bdfbbb50ef7e" (UID: "b03a628d-daed-477f-aa0d-bdfbbb50ef7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.562150 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.562123    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f6d5312-fdb6-40e6-80bb-703f052a0910" (UID: "6f6d5312-fdb6-40e6-80bb-703f052a0910"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.562455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.562228    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f6d5312-fdb6-40e6-80bb-703f052a0910" (UID: "6f6d5312-fdb6-40e6-80bb-703f052a0910"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.562455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.562289    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b03a628d-daed-477f-aa0d-bdfbbb50ef7e" (UID: "b03a628d-daed-477f-aa0d-bdfbbb50ef7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.564438 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.564409    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-dshm" (OuterVolumeSpecName: "dshm") pod "6f6d5312-fdb6-40e6-80bb-703f052a0910" (UID: "6f6d5312-fdb6-40e6-80bb-703f052a0910"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.564832 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.564801    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f6d5312-fdb6-40e6-80bb-703f052a0910-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f6d5312-fdb6-40e6-80bb-703f052a0910" (UID: "6f6d5312-fdb6-40e6-80bb-703f052a0910"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:58:50.565295 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.565273    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b03a628d-daed-477f-aa0d-bdfbbb50ef7e" (UID: "b03a628d-daed-477f-aa0d-bdfbbb50ef7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:58:50.566910 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.566272    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-kube-api-access-5m8xq" (OuterVolumeSpecName: "kube-api-access-5m8xq") pod "b03a628d-daed-477f-aa0d-bdfbbb50ef7e" (UID: "b03a628d-daed-477f-aa0d-bdfbbb50ef7e"). InnerVolumeSpecName "kube-api-access-5m8xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:58:50.566910 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.566280    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-dshm" (OuterVolumeSpecName: "dshm") pod "b03a628d-daed-477f-aa0d-bdfbbb50ef7e" (UID: "b03a628d-daed-477f-aa0d-bdfbbb50ef7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.567069 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.566999    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6d5312-fdb6-40e6-80bb-703f052a0910-kube-api-access-lcpmq" (OuterVolumeSpecName: "kube-api-access-lcpmq") pod "6f6d5312-fdb6-40e6-80bb-703f052a0910" (UID: "6f6d5312-fdb6-40e6-80bb-703f052a0910"). InnerVolumeSpecName "kube-api-access-lcpmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:58:50.571774 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.571665    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:58:50.662544 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662457    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-model-cache\") pod \"144c7178-a13b-4068-8866-5c0592344252\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") "
Sep 10 12:58:50.662544 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144c7178-a13b-4068-8866-5c0592344252-tls-certs\") pod \"144c7178-a13b-4068-8866-5c0592344252\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") "
Sep 10 12:58:50.662772 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662601    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-model-cache\") pod \"d941e342-015d-45b4-86eb-d0f2831388bf\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") "
Sep 10 12:58:50.662772 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662644    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-dshm\") pod \"144c7178-a13b-4068-8866-5c0592344252\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") "
Sep 10 12:58:50.662772 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662711    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-tmp-dir\") pod \"d941e342-015d-45b4-86eb-d0f2831388bf\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") "
Sep 10 12:58:50.662772 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662754    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-tmp-dir\") pod \"144c7178-a13b-4068-8866-5c0592344252\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") "
Sep 10 12:58:50.662994 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662812    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4prf\" (UniqueName: \"kubernetes.io/projected/144c7178-a13b-4068-8866-5c0592344252-kube-api-access-l4prf\") pod \"144c7178-a13b-4068-8866-5c0592344252\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") "
Sep 10 12:58:50.662994 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662858    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62526\" (UniqueName: \"kubernetes.io/projected/d941e342-015d-45b4-86eb-d0f2831388bf-kube-api-access-62526\") pod \"d941e342-015d-45b4-86eb-d0f2831388bf\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") "
Sep 10 12:58:50.662994 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662871    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "d941e342-015d-45b4-86eb-d0f2831388bf" (UID: "d941e342-015d-45b4-86eb-d0f2831388bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.662994 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662895    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-dshm\") pod \"d941e342-015d-45b4-86eb-d0f2831388bf\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") "
Sep 10 12:58:50.662994 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662920    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-home\") pod \"144c7178-a13b-4068-8866-5c0592344252\" (UID: \"144c7178-a13b-4068-8866-5c0592344252\") "
Sep 10 12:58:50.662994 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662928    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-model-cache" (OuterVolumeSpecName: "model-cache") pod "144c7178-a13b-4068-8866-5c0592344252" (UID: "144c7178-a13b-4068-8866-5c0592344252"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.662994 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662956    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d941e342-015d-45b4-86eb-d0f2831388bf-tls-certs\") pod \"d941e342-015d-45b4-86eb-d0f2831388bf\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") "
Sep 10 12:58:50.662994 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.662988    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-home\") pod \"d941e342-015d-45b4-86eb-d0f2831388bf\" (UID: \"d941e342-015d-45b4-86eb-d0f2831388bf\") "
Sep 10 12:58:50.663416 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663064    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "144c7178-a13b-4068-8866-5c0592344252" (UID: "144c7178-a13b-4068-8866-5c0592344252"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.663416 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663219    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d941e342-015d-45b4-86eb-d0f2831388bf" (UID: "d941e342-015d-45b4-86eb-d0f2831388bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.663416 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663351    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-home" (OuterVolumeSpecName: "home") pod "144c7178-a13b-4068-8866-5c0592344252" (UID: "144c7178-a13b-4068-8866-5c0592344252"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663430    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663452    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f6d5312-fdb6-40e6-80bb-703f052a0910-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663464    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663476    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663487    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663502    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663518    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663535    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m8xq\" (UniqueName: \"kubernetes.io/projected/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-kube-api-access-5m8xq\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663566    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663579    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663589    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663600    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663612    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcpmq\" (UniqueName: \"kubernetes.io/projected/6f6d5312-fdb6-40e6-80bb-703f052a0910-kube-api-access-lcpmq\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663627    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f6d5312-fdb6-40e6-80bb-703f052a0910-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.663636 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663641    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b03a628d-daed-477f-aa0d-bdfbbb50ef7e-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.664410 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.663842    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-home" (OuterVolumeSpecName: "home") pod "d941e342-015d-45b4-86eb-d0f2831388bf" (UID: "d941e342-015d-45b4-86eb-d0f2831388bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.665310 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.665280    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144c7178-a13b-4068-8866-5c0592344252-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "144c7178-a13b-4068-8866-5c0592344252" (UID: "144c7178-a13b-4068-8866-5c0592344252"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:58:50.665841 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.665812    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144c7178-a13b-4068-8866-5c0592344252-kube-api-access-l4prf" (OuterVolumeSpecName: "kube-api-access-l4prf") pod "144c7178-a13b-4068-8866-5c0592344252" (UID: "144c7178-a13b-4068-8866-5c0592344252"). InnerVolumeSpecName "kube-api-access-l4prf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:58:50.665841 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.665822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-dshm" (OuterVolumeSpecName: "dshm") pod "144c7178-a13b-4068-8866-5c0592344252" (UID: "144c7178-a13b-4068-8866-5c0592344252"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.665975 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.665953    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d941e342-015d-45b4-86eb-d0f2831388bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d941e342-015d-45b4-86eb-d0f2831388bf" (UID: "d941e342-015d-45b4-86eb-d0f2831388bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:58:50.666574 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.666536    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d941e342-015d-45b4-86eb-d0f2831388bf-kube-api-access-62526" (OuterVolumeSpecName: "kube-api-access-62526") pod "d941e342-015d-45b4-86eb-d0f2831388bf" (UID: "d941e342-015d-45b4-86eb-d0f2831388bf"). InnerVolumeSpecName "kube-api-access-62526". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:58:50.666666 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.666647    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-dshm" (OuterVolumeSpecName: "dshm") pod "d941e342-015d-45b4-86eb-d0f2831388bf" (UID: "d941e342-015d-45b4-86eb-d0f2831388bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:58:50.764529 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.764491    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4prf\" (UniqueName: \"kubernetes.io/projected/144c7178-a13b-4068-8866-5c0592344252-kube-api-access-l4prf\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.764529 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.764523    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62526\" (UniqueName: \"kubernetes.io/projected/d941e342-015d-45b4-86eb-d0f2831388bf-kube-api-access-62526\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.764529 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.764534    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.764856 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.764543    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.764856 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.764566    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d941e342-015d-45b4-86eb-d0f2831388bf-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.764856 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.764575    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d941e342-015d-45b4-86eb-d0f2831388bf-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.764856 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.764583    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144c7178-a13b-4068-8866-5c0592344252-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.764856 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.764590    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144c7178-a13b-4068-8866-5c0592344252-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:58:50.787242 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.787198    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"]
Sep 10 12:58:50.796728 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:50.796689    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/distinct-w4-1312ea-kserve-764d8f7746-69qfc"]
Sep 10 12:58:51.475777 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.475746    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"
Sep 10 12:58:51.476260 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.475734    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm" event={"ID":"d941e342-015d-45b4-86eb-d0f2831388bf","Type":"ContainerDied","Data":"b0506b1895f3471639c9373fe4ea99c99be40709f65b7aa642edadf26190641d"}
Sep 10 12:58:51.476260 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.475885    2563 scope.go:117] "RemoveContainer" containerID="e05ab99686a752d2c251e5441992996efa356f6e108618048ec21fe5d848b35e"
Sep 10 12:58:51.478513 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.478483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk" event={"ID":"144c7178-a13b-4068-8866-5c0592344252","Type":"ContainerDied","Data":"7c6bfd1ce9d452a2f771aa4878e4cf9d9de25517eca9cc605afe17b331f2a83a"}
Sep 10 12:58:51.478701 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.478597    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"
Sep 10 12:58:51.480079 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.480054    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579" event={"ID":"b03a628d-daed-477f-aa0d-bdfbbb50ef7e","Type":"ContainerDied","Data":"b6dda2bb01ac679e8548e971d2e0af9fb59a0393ba45e9390ff38b4106711aa1"}
Sep 10 12:58:51.480189 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.480061    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"
Sep 10 12:58:51.487647 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.487626    2563 scope.go:117] "RemoveContainer" containerID="f31160967042beea125449a8a5c4513456fb7986765d92fa8b830e807c2b8601"
Sep 10 12:58:51.495630 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.495593    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"]
Sep 10 12:58:51.497545 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.497525    2563 scope.go:117] "RemoveContainer" containerID="a7801efcff397bfb14b6deff56364afa3fa593ad40a4e4c432a91bf986efb4ce"
Sep 10 12:58:51.500153 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.500125    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/premium-w4-1312ea-kserve-78f9977467-789rm"]
Sep 10 12:58:51.514787 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.514755    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"]
Sep 10 12:58:51.521953 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.521921    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/distinct-2-w4-1312ea-kserve-69d7cf5b65-qv579"]
Sep 10 12:58:51.533692 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.533651    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"]
Sep 10 12:58:51.540352 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:51.540319    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-1312ea/simulator-w4-1312ea-kserve-5f558bb894-5ntdk"]
Sep 10 12:58:52.705728 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:52.705691    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="144c7178-a13b-4068-8866-5c0592344252" path="/var/lib/kubelet/pods/144c7178-a13b-4068-8866-5c0592344252/volumes"
Sep 10 12:58:52.706136 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:52.706121    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f6d5312-fdb6-40e6-80bb-703f052a0910" path="/var/lib/kubelet/pods/6f6d5312-fdb6-40e6-80bb-703f052a0910/volumes"
Sep 10 12:58:52.706482 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:52.706471    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b03a628d-daed-477f-aa0d-bdfbbb50ef7e" path="/var/lib/kubelet/pods/b03a628d-daed-477f-aa0d-bdfbbb50ef7e/volumes"
Sep 10 12:58:52.707164 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:58:52.707148    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d941e342-015d-45b4-86eb-d0f2831388bf" path="/var/lib/kubelet/pods/d941e342-015d-45b4-86eb-d0f2831388bf/volumes"
Sep 10 12:59:04.323790 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:04.323686    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"]
Sep 10 12:59:04.389150 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:04.389107    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"]
Sep 10 12:59:04.414016 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:04.413976    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"]
Sep 10 12:59:09.767984 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:09.767937    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"]
Sep 10 12:59:09.768414 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:09.768207    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy" containerID="cri-o://b481bbf836043fb5921a5706634c1254124d2e97ffa6a1dd55a62dff2d88fee1" gracePeriod=30
Sep 10 12:59:12.447640 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:12.447588    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 12:59:12.448040 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:12.447682    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 12:59:15.027075 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.027048    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:59:15.123874 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.123780    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-workload-certs\") pod \"6ade2931-49ff-4226-8c5d-05199d28dc0e\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") "
Sep 10 12:59:15.123874 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.123831    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-data\") pod \"6ade2931-49ff-4226-8c5d-05199d28dc0e\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") "
Sep 10 12:59:15.123874 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.123856    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-credential-socket\") pod \"6ade2931-49ff-4226-8c5d-05199d28dc0e\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") "
Sep 10 12:59:15.124199 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.123919    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4924\" (UniqueName: \"kubernetes.io/projected/6ade2931-49ff-4226-8c5d-05199d28dc0e-kube-api-access-v4924\") pod \"6ade2931-49ff-4226-8c5d-05199d28dc0e\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") "
Sep 10 12:59:15.124199 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.123974    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-envoy\") pod \"6ade2931-49ff-4226-8c5d-05199d28dc0e\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") "
Sep 10 12:59:15.124199 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.124007    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-token\") pod \"6ade2931-49ff-4226-8c5d-05199d28dc0e\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") "
Sep 10 12:59:15.124199 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.124043    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ade2931-49ff-4226-8c5d-05199d28dc0e-istiod-ca-cert\") pod \"6ade2931-49ff-4226-8c5d-05199d28dc0e\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") "
Sep 10 12:59:15.124199 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.124076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-podinfo\") pod \"6ade2931-49ff-4226-8c5d-05199d28dc0e\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") "
Sep 10 12:59:15.124199 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.124089    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ade2931-49ff-4226-8c5d-05199d28dc0e" (UID: "6ade2931-49ff-4226-8c5d-05199d28dc0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:15.124199 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.124105    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-workload-socket\") pod \"6ade2931-49ff-4226-8c5d-05199d28dc0e\" (UID: \"6ade2931-49ff-4226-8c5d-05199d28dc0e\") "
Sep 10 12:59:15.124611 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.124334    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ade2931-49ff-4226-8c5d-05199d28dc0e" (UID: "6ade2931-49ff-4226-8c5d-05199d28dc0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:15.124611 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.124438    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ade2931-49ff-4226-8c5d-05199d28dc0e" (UID: "6ade2931-49ff-4226-8c5d-05199d28dc0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:15.124611 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.124518    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ade2931-49ff-4226-8c5d-05199d28dc0e" (UID: "6ade2931-49ff-4226-8c5d-05199d28dc0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:15.124611 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.124590    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-workload-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:15.124611 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.124607    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-data\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:15.124890 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.124620    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-credential-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:15.124890 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.124753    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ade2931-49ff-4226-8c5d-05199d28dc0e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ade2931-49ff-4226-8c5d-05199d28dc0e" (UID: "6ade2931-49ff-4226-8c5d-05199d28dc0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:59:15.127393 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.127366    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ade2931-49ff-4226-8c5d-05199d28dc0e" (UID: "6ade2931-49ff-4226-8c5d-05199d28dc0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:15.127531 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.127416    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ade2931-49ff-4226-8c5d-05199d28dc0e" (UID: "6ade2931-49ff-4226-8c5d-05199d28dc0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:59:15.127632 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.127584    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ade2931-49ff-4226-8c5d-05199d28dc0e-kube-api-access-v4924" (OuterVolumeSpecName: "kube-api-access-v4924") pod "6ade2931-49ff-4226-8c5d-05199d28dc0e" (UID: "6ade2931-49ff-4226-8c5d-05199d28dc0e"). InnerVolumeSpecName "kube-api-access-v4924". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:59:15.127687 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.127617    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ade2931-49ff-4226-8c5d-05199d28dc0e" (UID: "6ade2931-49ff-4226-8c5d-05199d28dc0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:59:15.180941 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.180901    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5"]
Sep 10 12:59:15.186652 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.186624    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-30919a-29817405-ggwq5"]
Sep 10 12:59:15.225104 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.225069    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4924\" (UniqueName: \"kubernetes.io/projected/6ade2931-49ff-4226-8c5d-05199d28dc0e-kube-api-access-v4924\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:15.225104 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.225099    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-envoy\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:15.225279 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.225115    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-token\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:15.225279 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.225126    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ade2931-49ff-4226-8c5d-05199d28dc0e-istiod-ca-cert\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:15.225279 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.225137    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ade2931-49ff-4226-8c5d-05199d28dc0e-istio-podinfo\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:15.225279 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.225147    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ade2931-49ff-4226-8c5d-05199d28dc0e-workload-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:15.639053 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.639017    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerID="b481bbf836043fb5921a5706634c1254124d2e97ffa6a1dd55a62dff2d88fee1" exitCode=0
Sep 10 12:59:15.639221 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.639064    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"
Sep 10 12:59:15.639221 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.639100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" event={"ID":"6ade2931-49ff-4226-8c5d-05199d28dc0e","Type":"ContainerDied","Data":"b481bbf836043fb5921a5706634c1254124d2e97ffa6a1dd55a62dff2d88fee1"}
Sep 10 12:59:15.639221 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.639153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2" event={"ID":"6ade2931-49ff-4226-8c5d-05199d28dc0e","Type":"ContainerDied","Data":"5434c7b3d98626e3385fac62eba1897323ffe8eafa9686955bdb25a95e4a3a12"}
Sep 10 12:59:15.639221 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.639177    2563 scope.go:117] "RemoveContainer" containerID="b481bbf836043fb5921a5706634c1254124d2e97ffa6a1dd55a62dff2d88fee1"
Sep 10 12:59:15.649713 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.649690    2563 scope.go:117] "RemoveContainer" containerID="f9a349f61718cf6428542ddd2e42f7185fc134b34f45d06df5a3c50a7fc3a85e"
Sep 10 12:59:15.658072 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.658055    2563 scope.go:117] "RemoveContainer" containerID="b481bbf836043fb5921a5706634c1254124d2e97ffa6a1dd55a62dff2d88fee1"
Sep 10 12:59:15.658315 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:59:15.658298    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b481bbf836043fb5921a5706634c1254124d2e97ffa6a1dd55a62dff2d88fee1\": container with ID starting with b481bbf836043fb5921a5706634c1254124d2e97ffa6a1dd55a62dff2d88fee1 not found: ID does not exist" containerID="b481bbf836043fb5921a5706634c1254124d2e97ffa6a1dd55a62dff2d88fee1"
Sep 10 12:59:15.658364 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.658324    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b481bbf836043fb5921a5706634c1254124d2e97ffa6a1dd55a62dff2d88fee1"} err="failed to get container status \"b481bbf836043fb5921a5706634c1254124d2e97ffa6a1dd55a62dff2d88fee1\": rpc error: code = NotFound desc = could not find container \"b481bbf836043fb5921a5706634c1254124d2e97ffa6a1dd55a62dff2d88fee1\": container with ID starting with b481bbf836043fb5921a5706634c1254124d2e97ffa6a1dd55a62dff2d88fee1 not found: ID does not exist"
Sep 10 12:59:15.658364 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.658341    2563 scope.go:117] "RemoveContainer" containerID="f9a349f61718cf6428542ddd2e42f7185fc134b34f45d06df5a3c50a7fc3a85e"
Sep 10 12:59:15.658602 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:59:15.658579    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a349f61718cf6428542ddd2e42f7185fc134b34f45d06df5a3c50a7fc3a85e\": container with ID starting with f9a349f61718cf6428542ddd2e42f7185fc134b34f45d06df5a3c50a7fc3a85e not found: ID does not exist" containerID="f9a349f61718cf6428542ddd2e42f7185fc134b34f45d06df5a3c50a7fc3a85e"
Sep 10 12:59:15.658697 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.658608    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a349f61718cf6428542ddd2e42f7185fc134b34f45d06df5a3c50a7fc3a85e"} err="failed to get container status \"f9a349f61718cf6428542ddd2e42f7185fc134b34f45d06df5a3c50a7fc3a85e\": rpc error: code = NotFound desc = could not find container \"f9a349f61718cf6428542ddd2e42f7185fc134b34f45d06df5a3c50a7fc3a85e\": container with ID starting with f9a349f61718cf6428542ddd2e42f7185fc134b34f45d06df5a3c50a7fc3a85e not found: ID does not exist"
Sep 10 12:59:15.663585 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.663563    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"]
Sep 10 12:59:15.669137 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:15.669116    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-719d8a-openshift-default-79bb55bdbb-xmsr2"]
Sep 10 12:59:16.704869 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:16.704837    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c202854-fa69-421e-9bdc-a1458d773562" path="/var/lib/kubelet/pods/0c202854-fa69-421e-9bdc-a1458d773562/volumes"
Sep 10 12:59:16.705277 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:16.705262    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" path="/var/lib/kubelet/pods/6ade2931-49ff-4226-8c5d-05199d28dc0e/volumes"
Sep 10 12:59:19.346114 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.346067    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s" podUID="72a4386e-53a8-419a-be7e-9bb491d947b9" containerName="main" containerID="cri-o://a50aa57a8b8684a530b17aeb03e284d09e173176254236fffe92306a88561abd" gracePeriod=45
Sep 10 12:59:19.409232 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.409196    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk" podUID="4e63d3a2-3e29-41d2-924b-e87f8700d601" containerName="main" containerID="cri-o://3c63d510cb9791462366ee0c50db4c9eab7815e1a57cd75a94703eba16153d03" gracePeriod=45
Sep 10 12:59:19.434511 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.434479    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm" podUID="02441145-7b72-42f1-afba-7f87b9d3fd54" containerName="main" containerID="cri-o://12b31ae2ffd3b7865a608f2ec74ebb198984f5faae1aa86e2d22cbb153d2f750" gracePeriod=45
Sep 10 12:59:19.669360 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.669331    2563 generic.go:356] "Generic (PLEG): container finished" podID="02441145-7b72-42f1-afba-7f87b9d3fd54" containerID="12b31ae2ffd3b7865a608f2ec74ebb198984f5faae1aa86e2d22cbb153d2f750" exitCode=0
Sep 10 12:59:19.669451 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.669400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm" event={"ID":"02441145-7b72-42f1-afba-7f87b9d3fd54","Type":"ContainerDied","Data":"12b31ae2ffd3b7865a608f2ec74ebb198984f5faae1aa86e2d22cbb153d2f750"}
Sep 10 12:59:19.672069 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.672039    2563 generic.go:356] "Generic (PLEG): container finished" podID="72a4386e-53a8-419a-be7e-9bb491d947b9" containerID="a50aa57a8b8684a530b17aeb03e284d09e173176254236fffe92306a88561abd" exitCode=0
Sep 10 12:59:19.672173 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.672112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s" event={"ID":"72a4386e-53a8-419a-be7e-9bb491d947b9","Type":"ContainerDied","Data":"a50aa57a8b8684a530b17aeb03e284d09e173176254236fffe92306a88561abd"}
Sep 10 12:59:19.674195 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.674169    2563 generic.go:356] "Generic (PLEG): container finished" podID="4e63d3a2-3e29-41d2-924b-e87f8700d601" containerID="3c63d510cb9791462366ee0c50db4c9eab7815e1a57cd75a94703eba16153d03" exitCode=0
Sep 10 12:59:19.674294 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.674231    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk" event={"ID":"4e63d3a2-3e29-41d2-924b-e87f8700d601","Type":"ContainerDied","Data":"3c63d510cb9791462366ee0c50db4c9eab7815e1a57cd75a94703eba16153d03"}
Sep 10 12:59:19.696154 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.696124    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:59:19.756404 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.755064    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:59:19.761585 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.761537    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-home\") pod \"4e63d3a2-3e29-41d2-924b-e87f8700d601\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") "
Sep 10 12:59:19.761788 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.761615    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a4386e-53a8-419a-be7e-9bb491d947b9-tls-certs\") pod \"72a4386e-53a8-419a-be7e-9bb491d947b9\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") "
Sep 10 12:59:19.761788 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.761671    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-dshm\") pod \"72a4386e-53a8-419a-be7e-9bb491d947b9\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") "
Sep 10 12:59:19.761788 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.761689    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-model-cache\") pod \"72a4386e-53a8-419a-be7e-9bb491d947b9\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") "
Sep 10 12:59:19.761788 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.761724    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-home\") pod \"72a4386e-53a8-419a-be7e-9bb491d947b9\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") "
Sep 10 12:59:19.761788 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.761745    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-dshm\") pod \"4e63d3a2-3e29-41d2-924b-e87f8700d601\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") "
Sep 10 12:59:19.761788 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.761779    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e63d3a2-3e29-41d2-924b-e87f8700d601-tls-certs\") pod \"4e63d3a2-3e29-41d2-924b-e87f8700d601\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") "
Sep 10 12:59:19.762155 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.761811    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-tmp-dir\") pod \"4e63d3a2-3e29-41d2-924b-e87f8700d601\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") "
Sep 10 12:59:19.762155 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.761826    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-home" (OuterVolumeSpecName: "home") pod "4e63d3a2-3e29-41d2-924b-e87f8700d601" (UID: "4e63d3a2-3e29-41d2-924b-e87f8700d601"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.762155 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.761835    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7znw\" (UniqueName: \"kubernetes.io/projected/4e63d3a2-3e29-41d2-924b-e87f8700d601-kube-api-access-x7znw\") pod \"4e63d3a2-3e29-41d2-924b-e87f8700d601\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") "
Sep 10 12:59:19.762155 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.761899    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-model-cache\") pod \"4e63d3a2-3e29-41d2-924b-e87f8700d601\" (UID: \"4e63d3a2-3e29-41d2-924b-e87f8700d601\") "
Sep 10 12:59:19.762155 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.761936    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmmkf\" (UniqueName: \"kubernetes.io/projected/72a4386e-53a8-419a-be7e-9bb491d947b9-kube-api-access-cmmkf\") pod \"72a4386e-53a8-419a-be7e-9bb491d947b9\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") "
Sep 10 12:59:19.762155 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.761985    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-tmp-dir\") pod \"72a4386e-53a8-419a-be7e-9bb491d947b9\" (UID: \"72a4386e-53a8-419a-be7e-9bb491d947b9\") "
Sep 10 12:59:19.762455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.762177    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-home" (OuterVolumeSpecName: "home") pod "72a4386e-53a8-419a-be7e-9bb491d947b9" (UID: "72a4386e-53a8-419a-be7e-9bb491d947b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.762455 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.762380    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72a4386e-53a8-419a-be7e-9bb491d947b9" (UID: "72a4386e-53a8-419a-be7e-9bb491d947b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.762596 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.762519    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.762596 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.762538    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.762596 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.762582    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.763121 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.762789    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e63d3a2-3e29-41d2-924b-e87f8700d601" (UID: "4e63d3a2-3e29-41d2-924b-e87f8700d601"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.763121 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.762797    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "72a4386e-53a8-419a-be7e-9bb491d947b9" (UID: "72a4386e-53a8-419a-be7e-9bb491d947b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.763296 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.763228    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e63d3a2-3e29-41d2-924b-e87f8700d601" (UID: "4e63d3a2-3e29-41d2-924b-e87f8700d601"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.764960 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.764932    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-dshm" (OuterVolumeSpecName: "dshm") pod "72a4386e-53a8-419a-be7e-9bb491d947b9" (UID: "72a4386e-53a8-419a-be7e-9bb491d947b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.765211 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.765176    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e63d3a2-3e29-41d2-924b-e87f8700d601-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e63d3a2-3e29-41d2-924b-e87f8700d601" (UID: "4e63d3a2-3e29-41d2-924b-e87f8700d601"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:59:19.765313 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.765215    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-dshm" (OuterVolumeSpecName: "dshm") pod "4e63d3a2-3e29-41d2-924b-e87f8700d601" (UID: "4e63d3a2-3e29-41d2-924b-e87f8700d601"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.765405 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.765382    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:59:19.765405 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.765395    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a4386e-53a8-419a-be7e-9bb491d947b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72a4386e-53a8-419a-be7e-9bb491d947b9" (UID: "72a4386e-53a8-419a-be7e-9bb491d947b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:59:19.765606 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.765574    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e63d3a2-3e29-41d2-924b-e87f8700d601-kube-api-access-x7znw" (OuterVolumeSpecName: "kube-api-access-x7znw") pod "4e63d3a2-3e29-41d2-924b-e87f8700d601" (UID: "4e63d3a2-3e29-41d2-924b-e87f8700d601"). InnerVolumeSpecName "kube-api-access-x7znw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:59:19.766124 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.766099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a4386e-53a8-419a-be7e-9bb491d947b9-kube-api-access-cmmkf" (OuterVolumeSpecName: "kube-api-access-cmmkf") pod "72a4386e-53a8-419a-be7e-9bb491d947b9" (UID: "72a4386e-53a8-419a-be7e-9bb491d947b9"). InnerVolumeSpecName "kube-api-access-cmmkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:59:19.863753 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.863672    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-model-cache\") pod \"02441145-7b72-42f1-afba-7f87b9d3fd54\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") "
Sep 10 12:59:19.863753 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.863737    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02441145-7b72-42f1-afba-7f87b9d3fd54-tls-certs\") pod \"02441145-7b72-42f1-afba-7f87b9d3fd54\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") "
Sep 10 12:59:19.863919 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.863764    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-home\") pod \"02441145-7b72-42f1-afba-7f87b9d3fd54\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") "
Sep 10 12:59:19.863919 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.863804    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-dshm\") pod \"02441145-7b72-42f1-afba-7f87b9d3fd54\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") "
Sep 10 12:59:19.863919 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.863835    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsjq9\" (UniqueName: \"kubernetes.io/projected/02441145-7b72-42f1-afba-7f87b9d3fd54-kube-api-access-gsjq9\") pod \"02441145-7b72-42f1-afba-7f87b9d3fd54\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") "
Sep 10 12:59:19.863919 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.863861    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-tmp-dir\") pod \"02441145-7b72-42f1-afba-7f87b9d3fd54\" (UID: \"02441145-7b72-42f1-afba-7f87b9d3fd54\") "
Sep 10 12:59:19.864119 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.863976    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-model-cache" (OuterVolumeSpecName: "model-cache") pod "02441145-7b72-42f1-afba-7f87b9d3fd54" (UID: "02441145-7b72-42f1-afba-7f87b9d3fd54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.864119 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.864071    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-home" (OuterVolumeSpecName: "home") pod "02441145-7b72-42f1-afba-7f87b9d3fd54" (UID: "02441145-7b72-42f1-afba-7f87b9d3fd54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.864235 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.864210    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02441145-7b72-42f1-afba-7f87b9d3fd54" (UID: "02441145-7b72-42f1-afba-7f87b9d3fd54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.864289 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.864237    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.864289 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.864253    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72a4386e-53a8-419a-be7e-9bb491d947b9-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.864289 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.864263    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.864289 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.864270    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72a4386e-53a8-419a-be7e-9bb491d947b9-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.864289 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.864281    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.864497 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.864293    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e63d3a2-3e29-41d2-924b-e87f8700d601-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.864497 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.864303    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.864497 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.864310    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.864497 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.864320    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7znw\" (UniqueName: \"kubernetes.io/projected/4e63d3a2-3e29-41d2-924b-e87f8700d601-kube-api-access-x7znw\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.864497 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.864332    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e63d3a2-3e29-41d2-924b-e87f8700d601-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.864497 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.864345    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmmkf\" (UniqueName: \"kubernetes.io/projected/72a4386e-53a8-419a-be7e-9bb491d947b9-kube-api-access-cmmkf\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.865816 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.865799    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-dshm" (OuterVolumeSpecName: "dshm") pod "02441145-7b72-42f1-afba-7f87b9d3fd54" (UID: "02441145-7b72-42f1-afba-7f87b9d3fd54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:19.866270 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.866246    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02441145-7b72-42f1-afba-7f87b9d3fd54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02441145-7b72-42f1-afba-7f87b9d3fd54" (UID: "02441145-7b72-42f1-afba-7f87b9d3fd54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:59:19.866270 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.866260    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02441145-7b72-42f1-afba-7f87b9d3fd54-kube-api-access-gsjq9" (OuterVolumeSpecName: "kube-api-access-gsjq9") pod "02441145-7b72-42f1-afba-7f87b9d3fd54" (UID: "02441145-7b72-42f1-afba-7f87b9d3fd54"). InnerVolumeSpecName "kube-api-access-gsjq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:59:19.965656 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.965619    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02441145-7b72-42f1-afba-7f87b9d3fd54-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.965656 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.965649    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.965656 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.965658    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsjq9\" (UniqueName: \"kubernetes.io/projected/02441145-7b72-42f1-afba-7f87b9d3fd54-kube-api-access-gsjq9\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:19.965895 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:19.965666    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02441145-7b72-42f1-afba-7f87b9d3fd54-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:20.682230 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.682203    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"
Sep 10 12:59:20.682677 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.682228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s" event={"ID":"72a4386e-53a8-419a-be7e-9bb491d947b9","Type":"ContainerDied","Data":"f932f76571ea72af970c37ba57f53ce6777caa4d7e2b4b780b8652b2f5c76cd1"}
Sep 10 12:59:20.682677 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.682274    2563 scope.go:117] "RemoveContainer" containerID="a50aa57a8b8684a530b17aeb03e284d09e173176254236fffe92306a88561abd"
Sep 10 12:59:20.683849 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.683833    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"
Sep 10 12:59:20.683942 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.683871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk" event={"ID":"4e63d3a2-3e29-41d2-924b-e87f8700d601","Type":"ContainerDied","Data":"a48ba596d72d840fd6aa17fcda3e9b77b78d1f9ac63d220e214a9feef93f3f46"}
Sep 10 12:59:20.685432 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.685415    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"
Sep 10 12:59:20.685587 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.685448    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm" event={"ID":"02441145-7b72-42f1-afba-7f87b9d3fd54","Type":"ContainerDied","Data":"d66d6dd52fdb8adb15ffbaee7568e6a793b8da5c4117201e41fca8c1d05ebb02"}
Sep 10 12:59:20.693222 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.693206    2563 scope.go:117] "RemoveContainer" containerID="3c63d510cb9791462366ee0c50db4c9eab7815e1a57cd75a94703eba16153d03"
Sep 10 12:59:20.701563 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.701535    2563 scope.go:117] "RemoveContainer" containerID="12b31ae2ffd3b7865a608f2ec74ebb198984f5faae1aa86e2d22cbb153d2f750"
Sep 10 12:59:20.711913 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.711889    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"]
Sep 10 12:59:20.716194 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.716171    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/premium-w0-30919a-kserve-67f56cc8fc-kwl8s"]
Sep 10 12:59:20.728929 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.728896    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"]
Sep 10 12:59:20.733142 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.733120    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/unconfigured-w0-30919a-kserve-7c4574cbf9-zg2jm"]
Sep 10 12:59:20.744385 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.744338    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"]
Sep 10 12:59:20.746998 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:20.746975    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-30919a/simulator-w0-30919a-kserve-555c48d75b-gr9zk"]
Sep 10 12:59:21.360404 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:21.360369    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:59:22.704911 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:22.704871    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02441145-7b72-42f1-afba-7f87b9d3fd54" path="/var/lib/kubelet/pods/02441145-7b72-42f1-afba-7f87b9d3fd54/volumes"
Sep 10 12:59:22.705489 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:22.705466    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e63d3a2-3e29-41d2-924b-e87f8700d601" path="/var/lib/kubelet/pods/4e63d3a2-3e29-41d2-924b-e87f8700d601/volumes"
Sep 10 12:59:22.705923 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:22.705904    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72a4386e-53a8-419a-be7e-9bb491d947b9" path="/var/lib/kubelet/pods/72a4386e-53a8-419a-be7e-9bb491d947b9/volumes"
Sep 10 12:59:23.775270 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:23.775237    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"]
Sep 10 12:59:23.775667 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:23.775449    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerName="istio-proxy" containerID="cri-o://23d4773a142376af39019d1395576751bef5962061f73bcbbcd3611eb5b1a574" gracePeriod=30
Sep 10 12:59:26.881147 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:26.881115    2563 scope.go:117] "RemoveContainer" containerID="4dd16cc166fcd819906ac3ae9e4bafd1e8a26217bcffead4c813c9129703768e"
Sep 10 12:59:29.028349 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.028323    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:59:29.153597 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.153502    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-token\") pod \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") "
Sep 10 12:59:29.153597 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.153544    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-workload-socket\") pod \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") "
Sep 10 12:59:29.153597 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.153583    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-podinfo\") pod \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") "
Sep 10 12:59:29.153807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.153681    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istiod-ca-cert\") pod \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") "
Sep 10 12:59:29.153807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.153719    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-envoy\") pod \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") "
Sep 10 12:59:29.153807 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.153795    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxw9f\" (UniqueName: \"kubernetes.io/projected/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-kube-api-access-cxw9f\") pod \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") "
Sep 10 12:59:29.153970 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.153842    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-data\") pod \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") "
Sep 10 12:59:29.153970 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.153915    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-credential-socket\") pod \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") "
Sep 10 12:59:29.153970 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.153912    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f085c5f2-b9da-47e9-9342-7bd37db0b1f9" (UID: "f085c5f2-b9da-47e9-9342-7bd37db0b1f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:29.154132 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.153969    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-workload-certs\") pod \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\" (UID: \"f085c5f2-b9da-47e9-9342-7bd37db0b1f9\") "
Sep 10 12:59:29.154132 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.154029    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f085c5f2-b9da-47e9-9342-7bd37db0b1f9" (UID: "f085c5f2-b9da-47e9-9342-7bd37db0b1f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:59:29.154373 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.154130    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "f085c5f2-b9da-47e9-9342-7bd37db0b1f9" (UID: "f085c5f2-b9da-47e9-9342-7bd37db0b1f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:29.154373 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.154191    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f085c5f2-b9da-47e9-9342-7bd37db0b1f9" (UID: "f085c5f2-b9da-47e9-9342-7bd37db0b1f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:29.154373 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.154303    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f085c5f2-b9da-47e9-9342-7bd37db0b1f9" (UID: "f085c5f2-b9da-47e9-9342-7bd37db0b1f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:29.154539 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.154385    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-data\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:29.154539 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.154402    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-credential-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:29.154539 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.154416    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-workload-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:29.154539 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.154431    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-workload-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:29.154539 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.154443    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istiod-ca-cert\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:29.155853 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.155829    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f085c5f2-b9da-47e9-9342-7bd37db0b1f9" (UID: "f085c5f2-b9da-47e9-9342-7bd37db0b1f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:59:29.156035 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.156018    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f085c5f2-b9da-47e9-9342-7bd37db0b1f9" (UID: "f085c5f2-b9da-47e9-9342-7bd37db0b1f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:59:29.156353 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.156329    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-kube-api-access-cxw9f" (OuterVolumeSpecName: "kube-api-access-cxw9f") pod "f085c5f2-b9da-47e9-9342-7bd37db0b1f9" (UID: "f085c5f2-b9da-47e9-9342-7bd37db0b1f9"). InnerVolumeSpecName "kube-api-access-cxw9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:59:29.156447 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.156354    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "f085c5f2-b9da-47e9-9342-7bd37db0b1f9" (UID: "f085c5f2-b9da-47e9-9342-7bd37db0b1f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:59:29.255496 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.255458    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-token\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:29.255496 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.255489    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-podinfo\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:29.255496 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.255501    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-istio-envoy\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:29.255746 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.255518    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxw9f\" (UniqueName: \"kubernetes.io/projected/f085c5f2-b9da-47e9-9342-7bd37db0b1f9-kube-api-access-cxw9f\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 12:59:29.738073 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.738038    2563 generic.go:356] "Generic (PLEG): container finished" podID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerID="23d4773a142376af39019d1395576751bef5962061f73bcbbcd3611eb5b1a574" exitCode=0
Sep 10 12:59:29.738226 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.738089    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"
Sep 10 12:59:29.738226 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.738115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" event={"ID":"f085c5f2-b9da-47e9-9342-7bd37db0b1f9","Type":"ContainerDied","Data":"23d4773a142376af39019d1395576751bef5962061f73bcbbcd3611eb5b1a574"}
Sep 10 12:59:29.738226 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.738173    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx" event={"ID":"f085c5f2-b9da-47e9-9342-7bd37db0b1f9","Type":"ContainerDied","Data":"be00ccc6aa11aebebba22d369a48359f117cff4f72e47231724ecddb3852c637"}
Sep 10 12:59:29.738226 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.738186    2563 scope.go:117] "RemoveContainer" containerID="23d4773a142376af39019d1395576751bef5962061f73bcbbcd3611eb5b1a574"
Sep 10 12:59:29.748103 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.748085    2563 scope.go:117] "RemoveContainer" containerID="8ee4255861bd0688cdbd3f46ee1af8a44b7ef62574466fb9f10072d21f507857"
Sep 10 12:59:29.756490 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.756472    2563 scope.go:117] "RemoveContainer" containerID="23d4773a142376af39019d1395576751bef5962061f73bcbbcd3611eb5b1a574"
Sep 10 12:59:29.756748 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:59:29.756729    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d4773a142376af39019d1395576751bef5962061f73bcbbcd3611eb5b1a574\": container with ID starting with 23d4773a142376af39019d1395576751bef5962061f73bcbbcd3611eb5b1a574 not found: ID does not exist" containerID="23d4773a142376af39019d1395576751bef5962061f73bcbbcd3611eb5b1a574"
Sep 10 12:59:29.756805 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.756754    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d4773a142376af39019d1395576751bef5962061f73bcbbcd3611eb5b1a574"} err="failed to get container status \"23d4773a142376af39019d1395576751bef5962061f73bcbbcd3611eb5b1a574\": rpc error: code = NotFound desc = could not find container \"23d4773a142376af39019d1395576751bef5962061f73bcbbcd3611eb5b1a574\": container with ID starting with 23d4773a142376af39019d1395576751bef5962061f73bcbbcd3611eb5b1a574 not found: ID does not exist"
Sep 10 12:59:29.756805 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.756770    2563 scope.go:117] "RemoveContainer" containerID="8ee4255861bd0688cdbd3f46ee1af8a44b7ef62574466fb9f10072d21f507857"
Sep 10 12:59:29.757002 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:59:29.756979    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee4255861bd0688cdbd3f46ee1af8a44b7ef62574466fb9f10072d21f507857\": container with ID starting with 8ee4255861bd0688cdbd3f46ee1af8a44b7ef62574466fb9f10072d21f507857 not found: ID does not exist" containerID="8ee4255861bd0688cdbd3f46ee1af8a44b7ef62574466fb9f10072d21f507857"
Sep 10 12:59:29.757046 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.757008    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee4255861bd0688cdbd3f46ee1af8a44b7ef62574466fb9f10072d21f507857"} err="failed to get container status \"8ee4255861bd0688cdbd3f46ee1af8a44b7ef62574466fb9f10072d21f507857\": rpc error: code = NotFound desc = could not find container \"8ee4255861bd0688cdbd3f46ee1af8a44b7ef62574466fb9f10072d21f507857\": container with ID starting with 8ee4255861bd0688cdbd3f46ee1af8a44b7ef62574466fb9f10072d21f507857 not found: ID does not exist"
Sep 10 12:59:29.768490 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.768468    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"]
Sep 10 12:59:29.780024 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:29.780002    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-30919a-openshift-default-55776b64f8-zsfjx"]
Sep 10 12:59:30.704595 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:30.704542    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" path="/var/lib/kubelet/pods/f085c5f2-b9da-47e9-9342-7bd37db0b1f9/volumes"
Sep 10 12:59:37.430864 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:37.430828    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:59:50.725520 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:50.725480    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 12:59:52.373101 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373064    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7707-a03663"]
Sep 10 12:59:52.373578 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373546    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b03a628d-daed-477f-aa0d-bdfbbb50ef7e" containerName="main"
Sep 10 12:59:52.373578 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373577    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b03a628d-daed-477f-aa0d-bdfbbb50ef7e" containerName="main"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373586    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373592    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373602    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerName="istio-proxy"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373607    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerName="istio-proxy"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373614    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="144c7178-a13b-4068-8866-5c0592344252" containerName="main"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373620    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="144c7178-a13b-4068-8866-5c0592344252" containerName="main"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373629    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e63d3a2-3e29-41d2-924b-e87f8700d601" containerName="main"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373634    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e63d3a2-3e29-41d2-924b-e87f8700d601" containerName="main"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373641    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerName="istio-proxy"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373646    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerName="istio-proxy"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373657    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d941e342-015d-45b4-86eb-d0f2831388bf" containerName="main"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373662    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d941e342-015d-45b4-86eb-d0f2831388bf" containerName="main"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373668    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27637cc5-7e74-43fe-9300-b51fcfe9a40d" containerName="revoke-keys"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373673    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="27637cc5-7e74-43fe-9300-b51fcfe9a40d" containerName="revoke-keys"
Sep 10 12:59:52.373678 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373684    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f6d5312-fdb6-40e6-80bb-703f052a0910" containerName="main"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373689    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f6d5312-fdb6-40e6-80bb-703f052a0910" containerName="main"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373695    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02441145-7b72-42f1-afba-7f87b9d3fd54" containerName="main"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373701    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="02441145-7b72-42f1-afba-7f87b9d3fd54" containerName="main"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373714    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72a4386e-53a8-419a-be7e-9bb491d947b9" containerName="main"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373719    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="72a4386e-53a8-419a-be7e-9bb491d947b9" containerName="main"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373776    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b03a628d-daed-477f-aa0d-bdfbbb50ef7e" containerName="main"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373784    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f6d5312-fdb6-40e6-80bb-703f052a0910" containerName="main"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373790    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="02441145-7b72-42f1-afba-7f87b9d3fd54" containerName="main"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373796    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="72a4386e-53a8-419a-be7e-9bb491d947b9" containerName="main"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373802    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373807    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373812    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="27637cc5-7e74-43fe-9300-b51fcfe9a40d" containerName="revoke-keys"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373817    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="144c7178-a13b-4068-8866-5c0592344252" containerName="main"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373821    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerName="istio-proxy"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373826    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d941e342-015d-45b4-86eb-d0f2831388bf" containerName="main"
Sep 10 12:59:52.374168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.373831    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e63d3a2-3e29-41d2-924b-e87f8700d601" containerName="main"
Sep 10 12:59:52.378168 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.378142    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-a03663"
Sep 10 12:59:52.381853 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.381832    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-w7tsg\""
Sep 10 12:59:52.382247 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.382228    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-a03663"]
Sep 10 12:59:52.484942 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.484904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5plg\" (UniqueName: \"kubernetes.io/projected/36483a8c-c5fa-42dd-a73a-3e5c8f86e844-kube-api-access-f5plg\") pod \"test-curl-7707-a03663\" (UID: \"36483a8c-c5fa-42dd-a73a-3e5c8f86e844\") " pod="opendatahub/test-curl-7707-a03663"
Sep 10 12:59:52.585953 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.585918    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5plg\" (UniqueName: \"kubernetes.io/projected/36483a8c-c5fa-42dd-a73a-3e5c8f86e844-kube-api-access-f5plg\") pod \"test-curl-7707-a03663\" (UID: \"36483a8c-c5fa-42dd-a73a-3e5c8f86e844\") " pod="opendatahub/test-curl-7707-a03663"
Sep 10 12:59:52.596209 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.596169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5plg\" (UniqueName: \"kubernetes.io/projected/36483a8c-c5fa-42dd-a73a-3e5c8f86e844-kube-api-access-f5plg\") pod \"test-curl-7707-a03663\" (UID: \"36483a8c-c5fa-42dd-a73a-3e5c8f86e844\") " pod="opendatahub/test-curl-7707-a03663"
Sep 10 12:59:52.688765 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.688736    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-a03663"
Sep 10 12:59:52.963931 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:52.963907    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-a03663"]
Sep 10 12:59:52.966577 ip-10-0-133-150 kubenswrapper[2563]: W0910 12:59:52.966530    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36483a8c_c5fa_42dd_a73a_3e5c8f86e844.slice/crio-20bbc41dfbf75d049a6b5b84ef8c20e86df1506617c2402bdae0a0919789bbf5 WatchSource:0}: Error finding container 20bbc41dfbf75d049a6b5b84ef8c20e86df1506617c2402bdae0a0919789bbf5: Status 404 returned error can't find the container with id 20bbc41dfbf75d049a6b5b84ef8c20e86df1506617c2402bdae0a0919789bbf5
Sep 10 12:59:53.868323 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:53.868291    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-a03663" event={"ID":"36483a8c-c5fa-42dd-a73a-3e5c8f86e844","Type":"ContainerStarted","Data":"20bbc41dfbf75d049a6b5b84ef8c20e86df1506617c2402bdae0a0919789bbf5"}
Sep 10 12:59:55.883101 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:55.883055    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-a03663" event={"ID":"36483a8c-c5fa-42dd-a73a-3e5c8f86e844","Type":"ContainerStarted","Data":"06db05769bd3124d6f2757cadea260f560c81fdaf260a10fc3c94fa7aa567a36"}
Sep 10 12:59:55.906372 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:55.906319    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7707-a03663" podStartSLOduration=1.6725076890000001 podStartE2EDuration="3.906304164s" totalImagesPullingTime="2.233796475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:59:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:59:55.90507907 +0000 UTC m=+1769.829990559" watchObservedRunningTime="2026-09-10 12:59:55.906304164 +0000 UTC m=+1769.831215667"
Sep 10 12:59:56.607086 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:56.606989    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7707-a03663"]
Sep 10 12:59:56.610828 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:56.610801    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7707-a03663"]
Sep 10 12:59:56.888624 ip-10-0-133-150 kubenswrapper[2563]: E0910 12:59:56.888531    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7707-a03663" container="test-curl-7707-a03663"
Sep 10 12:59:57.893545 ip-10-0-133-150 kubenswrapper[2563]: I0910 12:59:57.893502    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7707-a03663" podUID="36483a8c-c5fa-42dd-a73a-3e5c8f86e844" containerName="test-curl-7707-a03663" containerID="cri-o://06db05769bd3124d6f2757cadea260f560c81fdaf260a10fc3c94fa7aa567a36" gracePeriod=2
Sep 10 13:00:00.034131 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.034106    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-a03663"
Sep 10 13:00:00.057430 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.057394    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5plg\" (UniqueName: \"kubernetes.io/projected/36483a8c-c5fa-42dd-a73a-3e5c8f86e844-kube-api-access-f5plg\") pod \"36483a8c-c5fa-42dd-a73a-3e5c8f86e844\" (UID: \"36483a8c-c5fa-42dd-a73a-3e5c8f86e844\") "
Sep 10 13:00:00.059534 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.059509    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36483a8c-c5fa-42dd-a73a-3e5c8f86e844-kube-api-access-f5plg" (OuterVolumeSpecName: "kube-api-access-f5plg") pod "36483a8c-c5fa-42dd-a73a-3e5c8f86e844" (UID: "36483a8c-c5fa-42dd-a73a-3e5c8f86e844"). InnerVolumeSpecName "kube-api-access-f5plg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:00:00.150700 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.150621    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817420-ft4wb"]
Sep 10 13:00:00.151046 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.151033    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36483a8c-c5fa-42dd-a73a-3e5c8f86e844" containerName="test-curl-7707-a03663"
Sep 10 13:00:00.151086 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.151047    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="36483a8c-c5fa-42dd-a73a-3e5c8f86e844" containerName="test-curl-7707-a03663"
Sep 10 13:00:00.151086 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.151058    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy"
Sep 10 13:00:00.151086 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.151064    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ade2931-49ff-4226-8c5d-05199d28dc0e" containerName="istio-proxy"
Sep 10 13:00:00.151186 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.151135    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="36483a8c-c5fa-42dd-a73a-3e5c8f86e844" containerName="test-curl-7707-a03663"
Sep 10 13:00:00.151186 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.151141    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f085c5f2-b9da-47e9-9342-7bd37db0b1f9" containerName="istio-proxy"
Sep 10 13:00:00.154114 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.154095    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817420-ft4wb"
Sep 10 13:00:00.156895 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.156877    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7c6kj\""
Sep 10 13:00:00.158764 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.158730    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5plg\" (UniqueName: \"kubernetes.io/projected/36483a8c-c5fa-42dd-a73a-3e5c8f86e844-kube-api-access-f5plg\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:00:00.165295 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.165275    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817420-ft4wb"
Sep 10 13:00:00.172754 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.168332    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817420-ft4wb"]
Sep 10 13:00:00.449873 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.449846    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817420-ft4wb"]
Sep 10 13:00:00.451997 ip-10-0-133-150 kubenswrapper[2563]: W0910 13:00:00.451959    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507dec08_5692_47e1_9c83_e632aed73127.slice/crio-8af26ecca6141fe3679e61d70d9014fdc42ae74ca508290e065d48068983325e WatchSource:0}: Error finding container 8af26ecca6141fe3679e61d70d9014fdc42ae74ca508290e065d48068983325e: Status 404 returned error can't find the container with id 8af26ecca6141fe3679e61d70d9014fdc42ae74ca508290e065d48068983325e
Sep 10 13:00:00.704613 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.704530    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36483a8c-c5fa-42dd-a73a-3e5c8f86e844" path="/var/lib/kubelet/pods/36483a8c-c5fa-42dd-a73a-3e5c8f86e844/volumes"
Sep 10 13:00:00.913976 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.913939    2563 generic.go:356] "Generic (PLEG): container finished" podID="36483a8c-c5fa-42dd-a73a-3e5c8f86e844" containerID="06db05769bd3124d6f2757cadea260f560c81fdaf260a10fc3c94fa7aa567a36" exitCode=137
Sep 10 13:00:00.914158 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.913983    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-a03663"
Sep 10 13:00:00.914158 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.914044    2563 scope.go:117] "RemoveContainer" containerID="06db05769bd3124d6f2757cadea260f560c81fdaf260a10fc3c94fa7aa567a36"
Sep 10 13:00:00.915931 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.915895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817420-ft4wb" event={"ID":"507dec08-5692-47e1-9c83-e632aed73127","Type":"ContainerStarted","Data":"cb2d68d88bc8bc6ccb3d262130e9bd7eb9fe174000c322d60b3d5db1704ebd70"}
Sep 10 13:00:00.916050 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.915955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817420-ft4wb" event={"ID":"507dec08-5692-47e1-9c83-e632aed73127","Type":"ContainerStarted","Data":"8af26ecca6141fe3679e61d70d9014fdc42ae74ca508290e065d48068983325e"}
Sep 10 13:00:00.923119 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.923103    2563 scope.go:117] "RemoveContainer" containerID="06db05769bd3124d6f2757cadea260f560c81fdaf260a10fc3c94fa7aa567a36"
Sep 10 13:00:00.923392 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:00:00.923365    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06db05769bd3124d6f2757cadea260f560c81fdaf260a10fc3c94fa7aa567a36\": container with ID starting with 06db05769bd3124d6f2757cadea260f560c81fdaf260a10fc3c94fa7aa567a36 not found: ID does not exist" containerID="06db05769bd3124d6f2757cadea260f560c81fdaf260a10fc3c94fa7aa567a36"
Sep 10 13:00:00.923443 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.923400    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06db05769bd3124d6f2757cadea260f560c81fdaf260a10fc3c94fa7aa567a36"} err="failed to get container status \"06db05769bd3124d6f2757cadea260f560c81fdaf260a10fc3c94fa7aa567a36\": rpc error: code = NotFound desc = could not find container \"06db05769bd3124d6f2757cadea260f560c81fdaf260a10fc3c94fa7aa567a36\": container with ID starting with 06db05769bd3124d6f2757cadea260f560c81fdaf260a10fc3c94fa7aa567a36 not found: ID does not exist"
Sep 10 13:00:00.947361 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:00.947307    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817420-ft4wb" podStartSLOduration=0.947290069 podStartE2EDuration="947.290069ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:00:00.946339116 +0000 UTC m=+1774.871250604" watchObservedRunningTime="2026-09-10 13:00:00.947290069 +0000 UTC m=+1774.872201556"
Sep 10 13:00:01.541802 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:01.541759    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:00:01.924532 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:01.924497    2563 generic.go:356] "Generic (PLEG): container finished" podID="507dec08-5692-47e1-9c83-e632aed73127" containerID="cb2d68d88bc8bc6ccb3d262130e9bd7eb9fe174000c322d60b3d5db1704ebd70" exitCode=0
Sep 10 13:00:01.924719 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:01.924531    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817420-ft4wb" event={"ID":"507dec08-5692-47e1-9c83-e632aed73127","Type":"ContainerDied","Data":"cb2d68d88bc8bc6ccb3d262130e9bd7eb9fe174000c322d60b3d5db1704ebd70"}
Sep 10 13:00:03.071947 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:03.071919    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817420-ft4wb"
Sep 10 13:00:03.936990 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:03.936963    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817420-ft4wb"
Sep 10 13:00:03.937164 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:03.937011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817420-ft4wb" event={"ID":"507dec08-5692-47e1-9c83-e632aed73127","Type":"ContainerDied","Data":"8af26ecca6141fe3679e61d70d9014fdc42ae74ca508290e065d48068983325e"}
Sep 10 13:00:03.937164 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:03.937055    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8af26ecca6141fe3679e61d70d9014fdc42ae74ca508290e065d48068983325e"
Sep 10 13:00:11.369098 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.369067    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"]
Sep 10 13:00:11.369612 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.369580    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="507dec08-5692-47e1-9c83-e632aed73127" containerName="cleanup"
Sep 10 13:00:11.369612 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.369595    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="507dec08-5692-47e1-9c83-e632aed73127" containerName="cleanup"
Sep 10 13:00:11.369704 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.369666    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="507dec08-5692-47e1-9c83-e632aed73127" containerName="cleanup"
Sep 10 13:00:11.372714 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.372694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.376388 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.376365    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c5adae\"/\"kube-root-ca.crt\""
Sep 10 13:00:11.377158 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.377135    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c5adae\"/\"sub-test-model-c5adae-kserve-self-signed-certs\""
Sep 10 13:00:11.377251 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.377166    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c5adae\"/\"default-dockercfg-gzt7q\""
Sep 10 13:00:11.377251 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.377142    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c5adae\"/\"openshift-service-ca.crt\""
Sep 10 13:00:11.392122 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.392099    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"]
Sep 10 13:00:11.466168 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.466126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-home\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.466346 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.466212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-model-cache\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.466346 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.466250    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsc7p\" (UniqueName: \"kubernetes.io/projected/eabcf4c9-763f-4ffe-8583-452329577c1a-kube-api-access-fsc7p\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.466346 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.466310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-tmp-dir\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.466478 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.466380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eabcf4c9-763f-4ffe-8583-452329577c1a-tls-certs\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.466478 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.466436    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-dshm\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.567161 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.567124    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eabcf4c9-763f-4ffe-8583-452329577c1a-tls-certs\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.567359 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.567171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-dshm\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.567359 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.567195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-home\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.567359 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.567232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-model-cache\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.567359 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.567253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsc7p\" (UniqueName: \"kubernetes.io/projected/eabcf4c9-763f-4ffe-8583-452329577c1a-kube-api-access-fsc7p\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.567359 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.567274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-tmp-dir\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.567804 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.567774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-home\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.567804 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.567790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-model-cache\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.567940 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.567833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-tmp-dir\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.569502 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.569471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-dshm\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.569828 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.569806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eabcf4c9-763f-4ffe-8583-452329577c1a-tls-certs\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.577057 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.577038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsc7p\" (UniqueName: \"kubernetes.io/projected/eabcf4c9-763f-4ffe-8583-452329577c1a-kube-api-access-fsc7p\") pod \"sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") " pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:11.682125 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:11.682091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:12.010010 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:12.009983    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"]
Sep 10 13:00:12.011177 ip-10-0-133-150 kubenswrapper[2563]: W0910 13:00:12.011151    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeabcf4c9_763f_4ffe_8583_452329577c1a.slice/crio-f95ddfff0465979467694655da06f08f260bdfa71e15e0cac01618e609be2a99 WatchSource:0}: Error finding container f95ddfff0465979467694655da06f08f260bdfa71e15e0cac01618e609be2a99: Status 404 returned error can't find the container with id f95ddfff0465979467694655da06f08f260bdfa71e15e0cac01618e609be2a99
Sep 10 13:00:12.991598 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:12.991563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj" event={"ID":"eabcf4c9-763f-4ffe-8583-452329577c1a","Type":"ContainerStarted","Data":"b2e0288c58aa084fd3baec05ca7040b436fbc31351502db4706cd368ecc1d261"}
Sep 10 13:00:12.991987 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:12.991610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj" event={"ID":"eabcf4c9-763f-4ffe-8583-452329577c1a","Type":"ContainerStarted","Data":"f95ddfff0465979467694655da06f08f260bdfa71e15e0cac01618e609be2a99"}
Sep 10 13:00:13.022018 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:13.021965    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj" podStartSLOduration=2.021948093 podStartE2EDuration="2.021948093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:00:13.020654245 +0000 UTC m=+1786.945565746" watchObservedRunningTime="2026-09-10 13:00:13.021948093 +0000 UTC m=+1786.946859580"
Sep 10 13:00:14.758562 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:14.758512    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:00:21.682285 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:21.682184    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:21.682285 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:21.682275    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:21.694946 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:21.694921    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:00:22.053654 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:00:22.053568    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:01:08.930329 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:08.930297    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"]
Sep 10 13:01:08.933664 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:08.933647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:08.936446 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:08.936417    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e5208b\"/\"kube-root-ca.crt\""
Sep 10 13:01:08.937319 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:08.937302    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e5208b\"/\"default-dockercfg-2lpg6\""
Sep 10 13:01:08.937623 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:08.937606    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e5208b\"/\"sub-test-model-e5208b-kserve-self-signed-certs\""
Sep 10 13:01:08.937703 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:08.937648    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e5208b\"/\"openshift-service-ca.crt\""
Sep 10 13:01:08.945214 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:08.945191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"]
Sep 10 13:01:09.031319 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.031281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-dshm\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.031319 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.031320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-home\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.031611 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.031346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj5w7\" (UniqueName: \"kubernetes.io/projected/fdaff800-1979-4a27-abd3-3a83f98983bc-kube-api-access-kj5w7\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.031611 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.031413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-model-cache\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.031611 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.031437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdaff800-1979-4a27-abd3-3a83f98983bc-tls-certs\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.031611 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.031461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-tmp-dir\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.132723 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.132683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-dshm\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.132927 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.132749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-home\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.132927 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.132789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj5w7\" (UniqueName: \"kubernetes.io/projected/fdaff800-1979-4a27-abd3-3a83f98983bc-kube-api-access-kj5w7\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.132927 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.132822    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-model-cache\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.132927 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.132861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdaff800-1979-4a27-abd3-3a83f98983bc-tls-certs\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.132927 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.132894    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-tmp-dir\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.133253 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.133227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-home\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.133326 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.133270    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-model-cache\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.133378 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.133323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-tmp-dir\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.135027 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.135008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-dshm\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.135294 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.135277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdaff800-1979-4a27-abd3-3a83f98983bc-tls-certs\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.142146 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.142117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj5w7\" (UniqueName: \"kubernetes.io/projected/fdaff800-1979-4a27-abd3-3a83f98983bc-kube-api-access-kj5w7\") pod \"sub-test-model-e5208b-kserve-d45ff58cf-d5jp7\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.244225 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.244124    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:09.523833 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:09.523804    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"]
Sep 10 13:01:09.525459 ip-10-0-133-150 kubenswrapper[2563]: W0910 13:01:09.525430    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdaff800_1979_4a27_abd3_3a83f98983bc.slice/crio-b16bff916a5a391eda3510e5c6f7ff45f6748514559a8b1df7aa9a6e2b683702 WatchSource:0}: Error finding container b16bff916a5a391eda3510e5c6f7ff45f6748514559a8b1df7aa9a6e2b683702: Status 404 returned error can't find the container with id b16bff916a5a391eda3510e5c6f7ff45f6748514559a8b1df7aa9a6e2b683702
Sep 10 13:01:10.301191 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:10.301154    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7" event={"ID":"fdaff800-1979-4a27-abd3-3a83f98983bc","Type":"ContainerStarted","Data":"3baaf788a5d4dfdbe29874e558237fa5d442e7a625c9d4bfef823b515beca841"}
Sep 10 13:01:10.301191 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:10.301193    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7" event={"ID":"fdaff800-1979-4a27-abd3-3a83f98983bc","Type":"ContainerStarted","Data":"b16bff916a5a391eda3510e5c6f7ff45f6748514559a8b1df7aa9a6e2b683702"}
Sep 10 13:01:10.323022 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:10.322965    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7" podStartSLOduration=2.322950304 podStartE2EDuration="2.322950304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:01:10.321463721 +0000 UTC m=+1844.246375210" watchObservedRunningTime="2026-09-10 13:01:10.322950304 +0000 UTC m=+1844.247861846"
Sep 10 13:01:12.637770 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:12.637734    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:01:19.244453 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:19.244401    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:19.244960 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:19.244467    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:19.257409 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:19.257382    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:19.365662 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:19.365634    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:01:33.367409 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:33.367367    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:01:46.873097 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:46.873061    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:01:54.387078 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:54.386993    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"]
Sep 10 13:01:55.282376 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:55.282341    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"]
Sep 10 13:01:57.835966 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:01:57.835929    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:02:08.526038 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:08.526004    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:02:09.406383 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.406343    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj" podUID="eabcf4c9-763f-4ffe-8583-452329577c1a" containerName="main" containerID="cri-o://b2e0288c58aa084fd3baec05ca7040b436fbc31351502db4706cd368ecc1d261" gracePeriod=45
Sep 10 13:02:09.643441 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.643408    2563 generic.go:356] "Generic (PLEG): container finished" podID="eabcf4c9-763f-4ffe-8583-452329577c1a" containerID="b2e0288c58aa084fd3baec05ca7040b436fbc31351502db4706cd368ecc1d261" exitCode=0
Sep 10 13:02:09.643793 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.643483    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj" event={"ID":"eabcf4c9-763f-4ffe-8583-452329577c1a","Type":"ContainerDied","Data":"b2e0288c58aa084fd3baec05ca7040b436fbc31351502db4706cd368ecc1d261"}
Sep 10 13:02:09.643793 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.643531    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj" event={"ID":"eabcf4c9-763f-4ffe-8583-452329577c1a","Type":"ContainerDied","Data":"f95ddfff0465979467694655da06f08f260bdfa71e15e0cac01618e609be2a99"}
Sep 10 13:02:09.643793 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.643576    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f95ddfff0465979467694655da06f08f260bdfa71e15e0cac01618e609be2a99"
Sep 10 13:02:09.655417 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.655395    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:02:09.725136 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.725106    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eabcf4c9-763f-4ffe-8583-452329577c1a-tls-certs\") pod \"eabcf4c9-763f-4ffe-8583-452329577c1a\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") "
Sep 10 13:02:09.725136 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.725141    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-dshm\") pod \"eabcf4c9-763f-4ffe-8583-452329577c1a\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") "
Sep 10 13:02:09.725388 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.725203    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-model-cache\") pod \"eabcf4c9-763f-4ffe-8583-452329577c1a\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") "
Sep 10 13:02:09.725388 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.725229    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-tmp-dir\") pod \"eabcf4c9-763f-4ffe-8583-452329577c1a\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") "
Sep 10 13:02:09.725388 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.725263    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsc7p\" (UniqueName: \"kubernetes.io/projected/eabcf4c9-763f-4ffe-8583-452329577c1a-kube-api-access-fsc7p\") pod \"eabcf4c9-763f-4ffe-8583-452329577c1a\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") "
Sep 10 13:02:09.725388 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.725305    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-home\") pod \"eabcf4c9-763f-4ffe-8583-452329577c1a\" (UID: \"eabcf4c9-763f-4ffe-8583-452329577c1a\") "
Sep 10 13:02:09.725612 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.725501    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-model-cache" (OuterVolumeSpecName: "model-cache") pod "eabcf4c9-763f-4ffe-8583-452329577c1a" (UID: "eabcf4c9-763f-4ffe-8583-452329577c1a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:02:09.725612 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.725496    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eabcf4c9-763f-4ffe-8583-452329577c1a" (UID: "eabcf4c9-763f-4ffe-8583-452329577c1a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:02:09.725733 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.725706    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-home" (OuterVolumeSpecName: "home") pod "eabcf4c9-763f-4ffe-8583-452329577c1a" (UID: "eabcf4c9-763f-4ffe-8583-452329577c1a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:02:09.725797 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.725779    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:02:09.725840 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.725806    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:02:09.727443 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.727413    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabcf4c9-763f-4ffe-8583-452329577c1a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eabcf4c9-763f-4ffe-8583-452329577c1a" (UID: "eabcf4c9-763f-4ffe-8583-452329577c1a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:02:09.727547 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.727467    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-dshm" (OuterVolumeSpecName: "dshm") pod "eabcf4c9-763f-4ffe-8583-452329577c1a" (UID: "eabcf4c9-763f-4ffe-8583-452329577c1a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:02:09.727547 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.727489    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabcf4c9-763f-4ffe-8583-452329577c1a-kube-api-access-fsc7p" (OuterVolumeSpecName: "kube-api-access-fsc7p") pod "eabcf4c9-763f-4ffe-8583-452329577c1a" (UID: "eabcf4c9-763f-4ffe-8583-452329577c1a"). InnerVolumeSpecName "kube-api-access-fsc7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:02:09.827280 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.827238    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:02:09.827280 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.827278    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eabcf4c9-763f-4ffe-8583-452329577c1a-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:02:09.827483 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.827292    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eabcf4c9-763f-4ffe-8583-452329577c1a-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:02:09.827483 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:09.827305    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsc7p\" (UniqueName: \"kubernetes.io/projected/eabcf4c9-763f-4ffe-8583-452329577c1a-kube-api-access-fsc7p\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:02:10.302070 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.302029    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7" podUID="fdaff800-1979-4a27-abd3-3a83f98983bc" containerName="main" containerID="cri-o://3baaf788a5d4dfdbe29874e558237fa5d442e7a625c9d4bfef823b515beca841" gracePeriod=45
Sep 10 13:02:10.559892 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.559830    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:02:10.637078 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.637047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj5w7\" (UniqueName: \"kubernetes.io/projected/fdaff800-1979-4a27-abd3-3a83f98983bc-kube-api-access-kj5w7\") pod \"fdaff800-1979-4a27-abd3-3a83f98983bc\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") "
Sep 10 13:02:10.637274 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.637117    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdaff800-1979-4a27-abd3-3a83f98983bc-tls-certs\") pod \"fdaff800-1979-4a27-abd3-3a83f98983bc\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") "
Sep 10 13:02:10.637274 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.637138    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-dshm\") pod \"fdaff800-1979-4a27-abd3-3a83f98983bc\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") "
Sep 10 13:02:10.637274 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.637170    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-tmp-dir\") pod \"fdaff800-1979-4a27-abd3-3a83f98983bc\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") "
Sep 10 13:02:10.637274 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.637221    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-home\") pod \"fdaff800-1979-4a27-abd3-3a83f98983bc\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") "
Sep 10 13:02:10.637274 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.637239    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-model-cache\") pod \"fdaff800-1979-4a27-abd3-3a83f98983bc\" (UID: \"fdaff800-1979-4a27-abd3-3a83f98983bc\") "
Sep 10 13:02:10.637573 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.637527    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdaff800-1979-4a27-abd3-3a83f98983bc" (UID: "fdaff800-1979-4a27-abd3-3a83f98983bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:02:10.637646 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.637510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-home" (OuterVolumeSpecName: "home") pod "fdaff800-1979-4a27-abd3-3a83f98983bc" (UID: "fdaff800-1979-4a27-abd3-3a83f98983bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:02:10.637646 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.637616    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdaff800-1979-4a27-abd3-3a83f98983bc" (UID: "fdaff800-1979-4a27-abd3-3a83f98983bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:02:10.639426 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.639402    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-dshm" (OuterVolumeSpecName: "dshm") pod "fdaff800-1979-4a27-abd3-3a83f98983bc" (UID: "fdaff800-1979-4a27-abd3-3a83f98983bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:02:10.639538 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.639405    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdaff800-1979-4a27-abd3-3a83f98983bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdaff800-1979-4a27-abd3-3a83f98983bc" (UID: "fdaff800-1979-4a27-abd3-3a83f98983bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:02:10.639538 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.639423    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdaff800-1979-4a27-abd3-3a83f98983bc-kube-api-access-kj5w7" (OuterVolumeSpecName: "kube-api-access-kj5w7") pod "fdaff800-1979-4a27-abd3-3a83f98983bc" (UID: "fdaff800-1979-4a27-abd3-3a83f98983bc"). InnerVolumeSpecName "kube-api-access-kj5w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:02:10.651141 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.651116    2563 generic.go:356] "Generic (PLEG): container finished" podID="fdaff800-1979-4a27-abd3-3a83f98983bc" containerID="3baaf788a5d4dfdbe29874e558237fa5d442e7a625c9d4bfef823b515beca841" exitCode=0
Sep 10 13:02:10.651537 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.651157    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"
Sep 10 13:02:10.651537 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.651204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7" event={"ID":"fdaff800-1979-4a27-abd3-3a83f98983bc","Type":"ContainerDied","Data":"3baaf788a5d4dfdbe29874e558237fa5d442e7a625c9d4bfef823b515beca841"}
Sep 10 13:02:10.651537 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.651254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7" event={"ID":"fdaff800-1979-4a27-abd3-3a83f98983bc","Type":"ContainerDied","Data":"b16bff916a5a391eda3510e5c6f7ff45f6748514559a8b1df7aa9a6e2b683702"}
Sep 10 13:02:10.651537 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.651276    2563 scope.go:117] "RemoveContainer" containerID="3baaf788a5d4dfdbe29874e558237fa5d442e7a625c9d4bfef823b515beca841"
Sep 10 13:02:10.651537 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.651347    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"
Sep 10 13:02:10.661904 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.661886    2563 scope.go:117] "RemoveContainer" containerID="3baaf788a5d4dfdbe29874e558237fa5d442e7a625c9d4bfef823b515beca841"
Sep 10 13:02:10.662312 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:02:10.662204    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3baaf788a5d4dfdbe29874e558237fa5d442e7a625c9d4bfef823b515beca841\": container with ID starting with 3baaf788a5d4dfdbe29874e558237fa5d442e7a625c9d4bfef823b515beca841 not found: ID does not exist" containerID="3baaf788a5d4dfdbe29874e558237fa5d442e7a625c9d4bfef823b515beca841"
Sep 10 13:02:10.662312 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.662293    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3baaf788a5d4dfdbe29874e558237fa5d442e7a625c9d4bfef823b515beca841"} err="failed to get container status \"3baaf788a5d4dfdbe29874e558237fa5d442e7a625c9d4bfef823b515beca841\": rpc error: code = NotFound desc = could not find container \"3baaf788a5d4dfdbe29874e558237fa5d442e7a625c9d4bfef823b515beca841\": container with ID starting with 3baaf788a5d4dfdbe29874e558237fa5d442e7a625c9d4bfef823b515beca841 not found: ID does not exist"
Sep 10 13:02:10.679024 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.679000    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"]
Sep 10 13:02:10.683453 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.683430    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/sub-test-model-e5208b-kserve-d45ff58cf-d5jp7"]
Sep 10 13:02:10.696380 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.696359    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"]
Sep 10 13:02:10.706089 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.706060    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdaff800-1979-4a27-abd3-3a83f98983bc" path="/var/lib/kubelet/pods/fdaff800-1979-4a27-abd3-3a83f98983bc/volumes"
Sep 10 13:02:10.706390 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.706371    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c5adae/sub-test-model-c5adae-kserve-5585b7bc7c-6kmrj"]
Sep 10 13:02:10.738540 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.738504    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdaff800-1979-4a27-abd3-3a83f98983bc-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:02:10.738540 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.738530    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:02:10.738540 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.738541    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:02:10.738757 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.738634    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:02:10.738757 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.738644    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdaff800-1979-4a27-abd3-3a83f98983bc-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:02:10.738757 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:10.738652    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj5w7\" (UniqueName: \"kubernetes.io/projected/fdaff800-1979-4a27-abd3-3a83f98983bc-kube-api-access-kj5w7\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:02:12.704901 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:12.704864    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eabcf4c9-763f-4ffe-8583-452329577c1a" path="/var/lib/kubelet/pods/eabcf4c9-763f-4ffe-8583-452329577c1a/volumes"
Sep 10 13:02:25.520763 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.520731    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"]
Sep 10 13:02:25.523178 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.521146    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eabcf4c9-763f-4ffe-8583-452329577c1a" containerName="main"
Sep 10 13:02:25.523178 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.521158    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="eabcf4c9-763f-4ffe-8583-452329577c1a" containerName="main"
Sep 10 13:02:25.523178 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.521182    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdaff800-1979-4a27-abd3-3a83f98983bc" containerName="main"
Sep 10 13:02:25.523178 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.521188    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdaff800-1979-4a27-abd3-3a83f98983bc" containerName="main"
Sep 10 13:02:25.523178 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.521256    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="eabcf4c9-763f-4ffe-8583-452329577c1a" containerName="main"
Sep 10 13:02:25.523178 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.521265    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdaff800-1979-4a27-abd3-3a83f98983bc" containerName="main"
Sep 10 13:02:25.524345 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.524326    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.527218 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.527198    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e5208b\"/\"openshift-service-ca.crt\""
Sep 10 13:02:25.527966 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.527948    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e5208b\"/\"default-dockercfg-2lpg6\""
Sep 10 13:02:25.528086 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.527988    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e5208b\"/\"kube-root-ca.crt\""
Sep 10 13:02:25.528151 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.528094    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e5208b\"/\"rate-test-model-e5208b-kserve-self-signed-certs\""
Sep 10 13:02:25.542560 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.542529    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"]
Sep 10 13:02:25.695694 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.695660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-tmp-dir\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.695886 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.695701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-model-cache\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.695886 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.695723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-home\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.695886 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.695831    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j2w2\" (UniqueName: \"kubernetes.io/projected/c10e1822-844c-4622-a186-51ab9eb38919-kube-api-access-5j2w2\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.696014 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.695890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-dshm\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.696088 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.696063    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e1822-844c-4622-a186-51ab9eb38919-tls-certs\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.797168 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.797061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-tmp-dir\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.797168 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.797105    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-model-cache\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.797168 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.797146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-home\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.797460 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.797184    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j2w2\" (UniqueName: \"kubernetes.io/projected/c10e1822-844c-4622-a186-51ab9eb38919-kube-api-access-5j2w2\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.797460 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.797239    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-dshm\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.797460 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.797314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e1822-844c-4622-a186-51ab9eb38919-tls-certs\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.797666 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.797513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-home\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.797666 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.797584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-model-cache\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.797776 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.797671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-tmp-dir\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.799644 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.799622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-dshm\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.799955 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.799937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e1822-844c-4622-a186-51ab9eb38919-tls-certs\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.816141 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.816089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j2w2\" (UniqueName: \"kubernetes.io/projected/c10e1822-844c-4622-a186-51ab9eb38919-kube-api-access-5j2w2\") pod \"rate-test-model-e5208b-kserve-7666c8bcc9-l76qb\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") " pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:25.833814 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:25.833787    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:26.111623 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:26.111594    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"]
Sep 10 13:02:26.113520 ip-10-0-133-150 kubenswrapper[2563]: W0910 13:02:26.113483    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc10e1822_844c_4622_a186_51ab9eb38919.slice/crio-8f08e1ec88f41345f65ddc32585df3521d79235fe2772c63acade8273efae167 WatchSource:0}: Error finding container 8f08e1ec88f41345f65ddc32585df3521d79235fe2772c63acade8273efae167: Status 404 returned error can't find the container with id 8f08e1ec88f41345f65ddc32585df3521d79235fe2772c63acade8273efae167
Sep 10 13:02:26.743956 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:26.743918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb" event={"ID":"c10e1822-844c-4622-a186-51ab9eb38919","Type":"ContainerStarted","Data":"949d2073fcea5a781bcee7851b22b6fcb9a6e9aad607673b1ece014e25c3002e"}
Sep 10 13:02:26.743956 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:26.743962    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb" event={"ID":"c10e1822-844c-4622-a186-51ab9eb38919","Type":"ContainerStarted","Data":"8f08e1ec88f41345f65ddc32585df3521d79235fe2772c63acade8273efae167"}
Sep 10 13:02:26.763008 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:26.762952    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb" podStartSLOduration=1.762933678 podStartE2EDuration="1.762933678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:02:26.761952673 +0000 UTC m=+1920.686864160" watchObservedRunningTime="2026-09-10 13:02:26.762933678 +0000 UTC m=+1920.687845166"
Sep 10 13:02:29.426134 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:29.426099    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:02:35.834335 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:35.834294    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:35.834738 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:35.834357    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:35.846827 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:35.846801    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:36.816771 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:36.816732    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:02:47.841436 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:47.841396    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:02:57.649862 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:02:57.649819    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:03:07.414565 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:07.414523    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:03:12.854476 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:12.854434    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"]
Sep 10 13:03:15.970467 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:15.970428    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:03:26.973334 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:26.973246    2563 scope.go:117] "RemoveContainer" containerID="e6fe26726eda4a4ba421528fe1033a07c12ba22b81845373e9bfdd1126c13a9a"
Sep 10 13:03:27.874927 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:27.874881    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb" podUID="c10e1822-844c-4622-a186-51ab9eb38919" containerName="main" containerID="cri-o://949d2073fcea5a781bcee7851b22b6fcb9a6e9aad607673b1ece014e25c3002e" gracePeriod=45
Sep 10 13:03:28.109791 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.109758    2563 generic.go:356] "Generic (PLEG): container finished" podID="c10e1822-844c-4622-a186-51ab9eb38919" containerID="949d2073fcea5a781bcee7851b22b6fcb9a6e9aad607673b1ece014e25c3002e" exitCode=0
Sep 10 13:03:28.110142 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.109792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb" event={"ID":"c10e1822-844c-4622-a186-51ab9eb38919","Type":"ContainerDied","Data":"949d2073fcea5a781bcee7851b22b6fcb9a6e9aad607673b1ece014e25c3002e"}
Sep 10 13:03:28.132178 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.132113    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:03:28.192998 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.192963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-model-cache\") pod \"c10e1822-844c-4622-a186-51ab9eb38919\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") "
Sep 10 13:03:28.193194 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.193023    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e1822-844c-4622-a186-51ab9eb38919-tls-certs\") pod \"c10e1822-844c-4622-a186-51ab9eb38919\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") "
Sep 10 13:03:28.193194 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.193047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j2w2\" (UniqueName: \"kubernetes.io/projected/c10e1822-844c-4622-a186-51ab9eb38919-kube-api-access-5j2w2\") pod \"c10e1822-844c-4622-a186-51ab9eb38919\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") "
Sep 10 13:03:28.193194 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.193094    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-tmp-dir\") pod \"c10e1822-844c-4622-a186-51ab9eb38919\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") "
Sep 10 13:03:28.193194 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.193133    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-dshm\") pod \"c10e1822-844c-4622-a186-51ab9eb38919\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") "
Sep 10 13:03:28.193437 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.193196    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-home\") pod \"c10e1822-844c-4622-a186-51ab9eb38919\" (UID: \"c10e1822-844c-4622-a186-51ab9eb38919\") "
Sep 10 13:03:28.193437 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.193326    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-model-cache" (OuterVolumeSpecName: "model-cache") pod "c10e1822-844c-4622-a186-51ab9eb38919" (UID: "c10e1822-844c-4622-a186-51ab9eb38919"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:03:28.193572 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.193535    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-home" (OuterVolumeSpecName: "home") pod "c10e1822-844c-4622-a186-51ab9eb38919" (UID: "c10e1822-844c-4622-a186-51ab9eb38919"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:03:28.193660 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.193638    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:03:28.193660 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.193653    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:03:28.193779 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.193725    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c10e1822-844c-4622-a186-51ab9eb38919" (UID: "c10e1822-844c-4622-a186-51ab9eb38919"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:03:28.195530 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.195491    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10e1822-844c-4622-a186-51ab9eb38919-kube-api-access-5j2w2" (OuterVolumeSpecName: "kube-api-access-5j2w2") pod "c10e1822-844c-4622-a186-51ab9eb38919" (UID: "c10e1822-844c-4622-a186-51ab9eb38919"). InnerVolumeSpecName "kube-api-access-5j2w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:03:28.195530 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.195510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10e1822-844c-4622-a186-51ab9eb38919-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c10e1822-844c-4622-a186-51ab9eb38919" (UID: "c10e1822-844c-4622-a186-51ab9eb38919"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:03:28.195711 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.195512    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-dshm" (OuterVolumeSpecName: "dshm") pod "c10e1822-844c-4622-a186-51ab9eb38919" (UID: "c10e1822-844c-4622-a186-51ab9eb38919"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:03:28.294388 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.294350    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:03:28.294388 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.294384    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c10e1822-844c-4622-a186-51ab9eb38919-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:03:28.294388 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.294393    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c10e1822-844c-4622-a186-51ab9eb38919-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:03:28.294739 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:28.294402    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j2w2\" (UniqueName: \"kubernetes.io/projected/c10e1822-844c-4622-a186-51ab9eb38919-kube-api-access-5j2w2\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:03:29.116668 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:29.116571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb" event={"ID":"c10e1822-844c-4622-a186-51ab9eb38919","Type":"ContainerDied","Data":"8f08e1ec88f41345f65ddc32585df3521d79235fe2772c63acade8273efae167"}
Sep 10 13:03:29.116668 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:29.116625    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"
Sep 10 13:03:29.117142 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:29.116631    2563 scope.go:117] "RemoveContainer" containerID="949d2073fcea5a781bcee7851b22b6fcb9a6e9aad607673b1ece014e25c3002e"
Sep 10 13:03:29.148827 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:29.148797    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"]
Sep 10 13:03:29.160818 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:29.160792    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e5208b/rate-test-model-e5208b-kserve-7666c8bcc9-l76qb"]
Sep 10 13:03:30.328865 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:30.328826    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:03:30.704957 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:30.704916    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c10e1822-844c-4622-a186-51ab9eb38919" path="/var/lib/kubelet/pods/c10e1822-844c-4622-a186-51ab9eb38919/volumes"
Sep 10 13:03:47.973390 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:47.973351    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"]
Sep 10 13:03:47.974038 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:47.974019    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10e1822-844c-4622-a186-51ab9eb38919" containerName="main"
Sep 10 13:03:47.974038 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:47.974038    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10e1822-844c-4622-a186-51ab9eb38919" containerName="main"
Sep 10 13:03:47.974198 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:47.974162    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c10e1822-844c-4622-a186-51ab9eb38919" containerName="main"
Sep 10 13:03:47.977961 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:47.977936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:47.982296 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:47.982273    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-6ea780-openshift-default-dockercfg-vqz5x\""
Sep 10 13:03:48.003256 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.003224    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"]
Sep 10 13:03:48.090520 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.090476    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-envoy\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.090723 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.090527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-workload-socket\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.090723 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.090636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-token\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.090814 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.090738    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-data\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.090814 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.090768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-podinfo\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.090814 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.090796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h6c9\" (UniqueName: \"kubernetes.io/projected/8de8aa95-a78e-485d-96cd-07b14b62a528-kube-api-access-2h6c9\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.090923 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.090846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-workload-certs\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.090923 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.090871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-credential-socket\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.090923 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.090896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de8aa95-a78e-485d-96cd-07b14b62a528-istiod-ca-cert\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.191832 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.191790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-token\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.192010 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.191877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-data\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.192010 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.191903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-podinfo\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.192010 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.191927    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6c9\" (UniqueName: \"kubernetes.io/projected/8de8aa95-a78e-485d-96cd-07b14b62a528-kube-api-access-2h6c9\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.192210 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.192055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-workload-certs\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.192210 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.192100    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-credential-socket\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.192210 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.192136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de8aa95-a78e-485d-96cd-07b14b62a528-istiod-ca-cert\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.192210 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.192171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-envoy\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.192210 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.192199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-workload-socket\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.192493 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.192443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-workload-certs\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.192573 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.192505    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-workload-socket\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.192670 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.192653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-data\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.192730 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.192693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-credential-socket\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.192940 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.192919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de8aa95-a78e-485d-96cd-07b14b62a528-istiod-ca-cert\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.194407 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.194383    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-envoy\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.194805 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.194789    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-podinfo\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.205137 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.205111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6c9\" (UniqueName: \"kubernetes.io/projected/8de8aa95-a78e-485d-96cd-07b14b62a528-kube-api-access-2h6c9\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.206648 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.206625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-token\") pod \"e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") " pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.291514 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.291424    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:48.618896 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.618868    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"]
Sep 10 13:03:48.620904 ip-10-0-133-150 kubenswrapper[2563]: W0910 13:03:48.620859    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de8aa95_a78e_485d_96cd_07b14b62a528.slice/crio-ff1bd861b667bbd9b4d76f2a6c6e0482d6af1002914228d9a5c497c03a3bf4d1 WatchSource:0}: Error finding container ff1bd861b667bbd9b4d76f2a6c6e0482d6af1002914228d9a5c497c03a3bf4d1: Status 404 returned error can't find the container with id ff1bd861b667bbd9b4d76f2a6c6e0482d6af1002914228d9a5c497c03a3bf4d1
Sep 10 13:03:48.623835 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.623790    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 13:03:48.623936 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.623883    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 13:03:48.623936 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:48.623924    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 13:03:49.233199 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:49.233159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl" event={"ID":"8de8aa95-a78e-485d-96cd-07b14b62a528","Type":"ContainerStarted","Data":"74a49fcd1c9367d074a153ee8e22031a22326b4165a9fc685b895c3340aeba8a"}
Sep 10 13:03:49.233736 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:49.233220    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl" event={"ID":"8de8aa95-a78e-485d-96cd-07b14b62a528","Type":"ContainerStarted","Data":"ff1bd861b667bbd9b4d76f2a6c6e0482d6af1002914228d9a5c497c03a3bf4d1"}
Sep 10 13:03:49.272226 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:49.272168    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl" podStartSLOduration=2.272147401 podStartE2EDuration="2.272147401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:03:49.266992629 +0000 UTC m=+2003.191904118" watchObservedRunningTime="2026-09-10 13:03:49.272147401 +0000 UTC m=+2003.197058890"
Sep 10 13:03:49.292082 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:49.292052    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:49.297360 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:49.297333    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:50.238471 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:50.238439    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:50.239380 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:50.239359    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:03:59.628993 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:03:59.628958    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:04:10.352922 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:10.352885    2563 generic.go:356] "Generic (PLEG): container finished" podID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerID="74a49fcd1c9367d074a153ee8e22031a22326b4165a9fc685b895c3340aeba8a" exitCode=0
Sep 10 13:04:10.353389 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:10.352976    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl" event={"ID":"8de8aa95-a78e-485d-96cd-07b14b62a528","Type":"ContainerDied","Data":"74a49fcd1c9367d074a153ee8e22031a22326b4165a9fc685b895c3340aeba8a"}
Sep 10 13:04:10.353583 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:10.353543    2563 scope.go:117] "RemoveContainer" containerID="74a49fcd1c9367d074a153ee8e22031a22326b4165a9fc685b895c3340aeba8a"
Sep 10 13:04:10.354844 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:10.354775    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 13:04:10.354913 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:10.354848    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 13:04:10.354913 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:10.354885    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 13:04:11.366911 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:11.366872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl" event={"ID":"8de8aa95-a78e-485d-96cd-07b14b62a528","Type":"ContainerStarted","Data":"b6e13fc4f6c149e2695029ed59a1253ee9de728d2068d3bd92687977d86b6c8e"}
Sep 10 13:04:12.291719 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:12.291680    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:04:12.291906 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:12.291877    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 10 13:04:12.292003 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:12.291953    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl" podUID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 10 13:04:13.291984 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:13.291946    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 10 13:04:13.292413 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:13.292005    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl" podUID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 10 13:04:14.292097 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:14.292057    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 10 13:04:14.292510 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:14.292115    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl" podUID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 10 13:04:15.291893 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:15.291854    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 10 13:04:15.292085 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:15.291915    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl" podUID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 10 13:04:16.292646 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:16.292605    2563 patch_prober.go:28] interesting pod/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Sep 10 13:04:16.293057 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:16.292667    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl" podUID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Sep 10 13:04:17.296681 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:17.296646    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:04:17.297117 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:17.297006    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:04:17.297722 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:17.297704    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:04:30.041006 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.040974    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"]
Sep 10 13:04:30.044521 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.044505    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.048017 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.047993    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c91a45\"/\"openshift-service-ca.crt\""
Sep 10 13:04:30.048017 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.048008    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c91a45\"/\"ipp-route-c91a45-kserve-self-signed-certs\""
Sep 10 13:04:30.048781 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.048767    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c91a45\"/\"default-dockercfg-lsnvx\""
Sep 10 13:04:30.048836 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.048800    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c91a45\"/\"kube-root-ca.crt\""
Sep 10 13:04:30.059081 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.059051    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"]
Sep 10 13:04:30.125327 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.125292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1816e32-03a9-47b2-8510-b4089f8ae2e3-tls-certs\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.125327 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.125332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-model-cache\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.125529 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.125355    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-tmp-dir\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.125529 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.125371    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-dshm\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.125529 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.125421    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-home\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.125529 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.125446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wflsv\" (UniqueName: \"kubernetes.io/projected/f1816e32-03a9-47b2-8510-b4089f8ae2e3-kube-api-access-wflsv\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.226835 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.226800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-model-cache\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.227022 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.226848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-tmp-dir\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.227022 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.226977    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-dshm\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.227150 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.227055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-home\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.227150 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.227095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wflsv\" (UniqueName: \"kubernetes.io/projected/f1816e32-03a9-47b2-8510-b4089f8ae2e3-kube-api-access-wflsv\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.227264 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.227239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-model-cache\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.227320 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.227265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1816e32-03a9-47b2-8510-b4089f8ae2e3-tls-certs\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.227320 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.227248    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-tmp-dir\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.227476 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.227454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-home\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.229097 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.229079    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-dshm\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.229664 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.229645    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1816e32-03a9-47b2-8510-b4089f8ae2e3-tls-certs\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.236474 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.236449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wflsv\" (UniqueName: \"kubernetes.io/projected/f1816e32-03a9-47b2-8510-b4089f8ae2e3-kube-api-access-wflsv\") pod \"ipp-route-c91a45-kserve-54765dcddc-w7wpd\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") " pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.354573 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.354451    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:30.640718 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:30.640689    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"]
Sep 10 13:04:30.641162 ip-10-0-133-150 kubenswrapper[2563]: W0910 13:04:30.641134    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1816e32_03a9_47b2_8510_b4089f8ae2e3.slice/crio-2c0ed1186d198931f8c6d6220c15b815b00723421f41480020a4f7c37c5d3e43 WatchSource:0}: Error finding container 2c0ed1186d198931f8c6d6220c15b815b00723421f41480020a4f7c37c5d3e43: Status 404 returned error can't find the container with id 2c0ed1186d198931f8c6d6220c15b815b00723421f41480020a4f7c37c5d3e43
Sep 10 13:04:31.477993 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:31.477957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd" event={"ID":"f1816e32-03a9-47b2-8510-b4089f8ae2e3","Type":"ContainerStarted","Data":"fd7008b829654f3bbcfa9458d8490a7f7f06de584740b41b74d9342777c3a282"}
Sep 10 13:04:31.478395 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:31.478003    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd" event={"ID":"f1816e32-03a9-47b2-8510-b4089f8ae2e3","Type":"ContainerStarted","Data":"2c0ed1186d198931f8c6d6220c15b815b00723421f41480020a4f7c37c5d3e43"}
Sep 10 13:04:31.505269 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:31.505219    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd" podStartSLOduration=1.5052023490000002 podStartE2EDuration="1.505202349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:04:31.503743739 +0000 UTC m=+2045.428655228" watchObservedRunningTime="2026-09-10 13:04:31.505202349 +0000 UTC m=+2045.430113836"
Sep 10 13:04:34.153160 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:34.153119    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:04:40.355183 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:40.355143    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:40.355725 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:40.355200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:40.368043 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:40.368014    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:40.545997 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:40.545959    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:04:48.990434 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:48.990352    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:04:56.198126 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.194685    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"]
Sep 10 13:04:56.200343 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.200313    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.208082 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.208058    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-c8b83b-openshift-default-dockercfg-zr5fv\""
Sep 10 13:04:56.290084 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.290041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-workload-socket\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.290257 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.290090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5pb\" (UniqueName: \"kubernetes.io/projected/7be809a2-426b-402f-95f0-068f302a38be-kube-api-access-sq5pb\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.290257 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.290122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-credential-socket\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.290257 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.290181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be809a2-426b-402f-95f0-068f302a38be-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.290257 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.290251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-istio-data\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.290404 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.290282    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-workload-certs\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.290404 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.290320    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7be809a2-426b-402f-95f0-068f302a38be-istio-token\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.290404 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.290352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-istio-envoy\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.290501 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.290414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be809a2-426b-402f-95f0-068f302a38be-istio-podinfo\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.328921 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.328879    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"]
Sep 10 13:04:56.391006 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.390967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be809a2-426b-402f-95f0-068f302a38be-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.391182 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.391027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-istio-data\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.391182 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.391050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-workload-certs\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.391182 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.391081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7be809a2-426b-402f-95f0-068f302a38be-istio-token\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.391182 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.391109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-istio-envoy\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.391182 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.391164    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be809a2-426b-402f-95f0-068f302a38be-istio-podinfo\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.391500 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.391199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-workload-socket\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.391500 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.391223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5pb\" (UniqueName: \"kubernetes.io/projected/7be809a2-426b-402f-95f0-068f302a38be-kube-api-access-sq5pb\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.391500 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.391255    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-credential-socket\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.391708 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.391606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-istio-data\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.391708 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.391693    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-workload-certs\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.391914 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.391879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be809a2-426b-402f-95f0-068f302a38be-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.392009 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.391884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-workload-socket\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.392009 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.391886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-credential-socket\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.393794 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.393772    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-istio-envoy\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.393997 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.393981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be809a2-426b-402f-95f0-068f302a38be-istio-podinfo\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.409275 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.409245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7be809a2-426b-402f-95f0-068f302a38be-istio-token\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.417695 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.417672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5pb\" (UniqueName: \"kubernetes.io/projected/7be809a2-426b-402f-95f0-068f302a38be-kube-api-access-sq5pb\") pod \"e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") " pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.512190 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.512089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:56.839458 ip-10-0-133-150 kubenswrapper[2563]: W0910 13:04:56.839426    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be809a2_426b_402f_95f0_068f302a38be.slice/crio-62a580c09eae4b859f9d58f7e724893c3b19beea7d1229b5556934e43c12b2ef WatchSource:0}: Error finding container 62a580c09eae4b859f9d58f7e724893c3b19beea7d1229b5556934e43c12b2ef: Status 404 returned error can't find the container with id 62a580c09eae4b859f9d58f7e724893c3b19beea7d1229b5556934e43c12b2ef
Sep 10 13:04:56.840594 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.840569    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"]
Sep 10 13:04:56.841763 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.841730    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 13:04:56.841854 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.841795    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 13:04:56.841854 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:56.841821    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 13:04:57.633131 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:57.633089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" event={"ID":"7be809a2-426b-402f-95f0-068f302a38be","Type":"ContainerStarted","Data":"b4a5b355356433077dd8d61ec7dec848b459dadbaaf8ef73f84ababd62df5524"}
Sep 10 13:04:57.633530 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:57.633158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" event={"ID":"7be809a2-426b-402f-95f0-068f302a38be","Type":"ContainerStarted","Data":"62a580c09eae4b859f9d58f7e724893c3b19beea7d1229b5556934e43c12b2ef"}
Sep 10 13:04:57.675528 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:57.675461    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" podStartSLOduration=1.675437646 podStartE2EDuration="1.675437646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:04:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:04:57.673294599 +0000 UTC m=+2071.598206090" watchObservedRunningTime="2026-09-10 13:04:57.675437646 +0000 UTC m=+2071.600349135"
Sep 10 13:04:58.512473 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:58.512433    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:58.518062 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:58.518035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:58.638769 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:58.638734    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:04:58.639658 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:04:58.639634    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:05:11.110972 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:11.110929    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:05:21.785845 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:21.785809    2563 generic.go:356] "Generic (PLEG): container finished" podID="7be809a2-426b-402f-95f0-068f302a38be" containerID="b4a5b355356433077dd8d61ec7dec848b459dadbaaf8ef73f84ababd62df5524" exitCode=0
Sep 10 13:05:21.786262 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:21.785882    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" event={"ID":"7be809a2-426b-402f-95f0-068f302a38be","Type":"ContainerDied","Data":"b4a5b355356433077dd8d61ec7dec848b459dadbaaf8ef73f84ababd62df5524"}
Sep 10 13:05:21.786335 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:21.786321    2563 scope.go:117] "RemoveContainer" containerID="b4a5b355356433077dd8d61ec7dec848b459dadbaaf8ef73f84ababd62df5524"
Sep 10 13:05:21.787592 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:21.787566    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 13:05:21.787647 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:21.787637    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 13:05:21.787683 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:21.787668    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 10 13:05:22.793809 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:22.793772    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" event={"ID":"7be809a2-426b-402f-95f0-068f302a38be","Type":"ContainerStarted","Data":"ac46742f4c5b5819694d73d3eb55aa0598839b1b57e542c24ecdad812e10e9cc"}
Sep 10 13:05:23.513099 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:23.513061    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:05:23.513300 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:23.513278    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 10 13:05:23.513348 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:23.513334    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" podUID="7be809a2-426b-402f-95f0-068f302a38be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 10 13:05:24.513434 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:24.513388    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 10 13:05:24.513966 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:24.513454    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" podUID="7be809a2-426b-402f-95f0-068f302a38be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 10 13:05:25.513434 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:25.513394    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 10 13:05:25.513634 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:25.513450    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" podUID="7be809a2-426b-402f-95f0-068f302a38be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 10 13:05:26.513062 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:26.513018    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 10 13:05:26.513251 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:26.513076    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" podUID="7be809a2-426b-402f-95f0-068f302a38be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 10 13:05:27.007409 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:27.007378    2563 scope.go:117] "RemoveContainer" containerID="c1e8e51d082246dfd7fce283464c74f48647614bed57af9acb83290be52819de"
Sep 10 13:05:27.513093 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:27.513051    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Sep 10 13:05:27.513279 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:27.513119    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" podUID="7be809a2-426b-402f-95f0-068f302a38be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Sep 10 13:05:28.516784 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:28.516755    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:05:28.517187 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:28.517111    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:05:28.517880 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:28.517859    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:05:48.359245 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:48.359212    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:05:53.290413 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:53.290376    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"]
Sep 10 13:05:53.290857 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:53.290587    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" podUID="7be809a2-426b-402f-95f0-068f302a38be" containerName="istio-proxy" containerID="cri-o://ac46742f4c5b5819694d73d3eb55aa0598839b1b57e542c24ecdad812e10e9cc" gracePeriod=30
Sep 10 13:05:58.541691 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.541661    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:05:58.633829 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.633734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-credential-socket\") pod \"7be809a2-426b-402f-95f0-068f302a38be\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") "
Sep 10 13:05:58.633829 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.633803    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-istio-data\") pod \"7be809a2-426b-402f-95f0-068f302a38be\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") "
Sep 10 13:05:58.634062 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.633837    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-workload-certs\") pod \"7be809a2-426b-402f-95f0-068f302a38be\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") "
Sep 10 13:05:58.634062 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.633863    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be809a2-426b-402f-95f0-068f302a38be-istiod-ca-cert\") pod \"7be809a2-426b-402f-95f0-068f302a38be\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") "
Sep 10 13:05:58.634062 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.633893    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq5pb\" (UniqueName: \"kubernetes.io/projected/7be809a2-426b-402f-95f0-068f302a38be-kube-api-access-sq5pb\") pod \"7be809a2-426b-402f-95f0-068f302a38be\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") "
Sep 10 13:05:58.634062 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.633924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be809a2-426b-402f-95f0-068f302a38be-istio-podinfo\") pod \"7be809a2-426b-402f-95f0-068f302a38be\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") "
Sep 10 13:05:58.634062 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.633950    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-istio-envoy\") pod \"7be809a2-426b-402f-95f0-068f302a38be\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") "
Sep 10 13:05:58.634062 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.633996    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7be809a2-426b-402f-95f0-068f302a38be-istio-token\") pod \"7be809a2-426b-402f-95f0-068f302a38be\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") "
Sep 10 13:05:58.634062 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.634045    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7be809a2-426b-402f-95f0-068f302a38be" (UID: "7be809a2-426b-402f-95f0-068f302a38be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:05:58.634413 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.634061    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-workload-socket\") pod \"7be809a2-426b-402f-95f0-068f302a38be\" (UID: \"7be809a2-426b-402f-95f0-068f302a38be\") "
Sep 10 13:05:58.634413 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.634112    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-istio-data" (OuterVolumeSpecName: "istio-data") pod "7be809a2-426b-402f-95f0-068f302a38be" (UID: "7be809a2-426b-402f-95f0-068f302a38be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:05:58.634413 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.634181    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7be809a2-426b-402f-95f0-068f302a38be" (UID: "7be809a2-426b-402f-95f0-068f302a38be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:05:58.634413 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.634315    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be809a2-426b-402f-95f0-068f302a38be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7be809a2-426b-402f-95f0-068f302a38be" (UID: "7be809a2-426b-402f-95f0-068f302a38be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:05:58.634413 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.634318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7be809a2-426b-402f-95f0-068f302a38be" (UID: "7be809a2-426b-402f-95f0-068f302a38be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:05:58.634620 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.634509    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-credential-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:05:58.634620 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.634525    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-istio-data\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:05:58.634620 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.634539    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-workload-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:05:58.634620 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.634575    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7be809a2-426b-402f-95f0-068f302a38be-istiod-ca-cert\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:05:58.634620 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.634588    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-workload-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:05:58.636323 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.636297    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7be809a2-426b-402f-95f0-068f302a38be" (UID: "7be809a2-426b-402f-95f0-068f302a38be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:05:58.636436 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.636338    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be809a2-426b-402f-95f0-068f302a38be-kube-api-access-sq5pb" (OuterVolumeSpecName: "kube-api-access-sq5pb") pod "7be809a2-426b-402f-95f0-068f302a38be" (UID: "7be809a2-426b-402f-95f0-068f302a38be"). InnerVolumeSpecName "kube-api-access-sq5pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:05:58.636636 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.636617    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be809a2-426b-402f-95f0-068f302a38be-istio-token" (OuterVolumeSpecName: "istio-token") pod "7be809a2-426b-402f-95f0-068f302a38be" (UID: "7be809a2-426b-402f-95f0-068f302a38be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:05:58.636727 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.636634    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7be809a2-426b-402f-95f0-068f302a38be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7be809a2-426b-402f-95f0-068f302a38be" (UID: "7be809a2-426b-402f-95f0-068f302a38be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:05:58.735338 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.735310    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq5pb\" (UniqueName: \"kubernetes.io/projected/7be809a2-426b-402f-95f0-068f302a38be-kube-api-access-sq5pb\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:05:58.735338 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.735335    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7be809a2-426b-402f-95f0-068f302a38be-istio-podinfo\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:05:58.735497 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.735345    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7be809a2-426b-402f-95f0-068f302a38be-istio-envoy\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:05:58.735497 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:58.735354    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7be809a2-426b-402f-95f0-068f302a38be-istio-token\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:05:59.005002 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.004962    2563 generic.go:356] "Generic (PLEG): container finished" podID="7be809a2-426b-402f-95f0-068f302a38be" containerID="ac46742f4c5b5819694d73d3eb55aa0598839b1b57e542c24ecdad812e10e9cc" exitCode=0
Sep 10 13:05:59.005002 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.005004    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"
Sep 10 13:05:59.005251 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.005017    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" event={"ID":"7be809a2-426b-402f-95f0-068f302a38be","Type":"ContainerDied","Data":"ac46742f4c5b5819694d73d3eb55aa0598839b1b57e542c24ecdad812e10e9cc"}
Sep 10 13:05:59.005251 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.005070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" event={"ID":"7be809a2-426b-402f-95f0-068f302a38be","Type":"ContainerDied","Data":"62a580c09eae4b859f9d58f7e724893c3b19beea7d1229b5556934e43c12b2ef"}
Sep 10 13:05:59.005251 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.005094    2563 scope.go:117] "RemoveContainer" containerID="ac46742f4c5b5819694d73d3eb55aa0598839b1b57e542c24ecdad812e10e9cc"
Sep 10 13:05:59.014431 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.014409    2563 scope.go:117] "RemoveContainer" containerID="b4a5b355356433077dd8d61ec7dec848b459dadbaaf8ef73f84ababd62df5524"
Sep 10 13:05:59.023151 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.023131    2563 scope.go:117] "RemoveContainer" containerID="ac46742f4c5b5819694d73d3eb55aa0598839b1b57e542c24ecdad812e10e9cc"
Sep 10 13:05:59.023435 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:05:59.023416    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac46742f4c5b5819694d73d3eb55aa0598839b1b57e542c24ecdad812e10e9cc\": container with ID starting with ac46742f4c5b5819694d73d3eb55aa0598839b1b57e542c24ecdad812e10e9cc not found: ID does not exist" containerID="ac46742f4c5b5819694d73d3eb55aa0598839b1b57e542c24ecdad812e10e9cc"
Sep 10 13:05:59.023485 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.023444    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac46742f4c5b5819694d73d3eb55aa0598839b1b57e542c24ecdad812e10e9cc"} err="failed to get container status \"ac46742f4c5b5819694d73d3eb55aa0598839b1b57e542c24ecdad812e10e9cc\": rpc error: code = NotFound desc = could not find container \"ac46742f4c5b5819694d73d3eb55aa0598839b1b57e542c24ecdad812e10e9cc\": container with ID starting with ac46742f4c5b5819694d73d3eb55aa0598839b1b57e542c24ecdad812e10e9cc not found: ID does not exist"
Sep 10 13:05:59.023485 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.023463    2563 scope.go:117] "RemoveContainer" containerID="b4a5b355356433077dd8d61ec7dec848b459dadbaaf8ef73f84ababd62df5524"
Sep 10 13:05:59.023725 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:05:59.023710    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a5b355356433077dd8d61ec7dec848b459dadbaaf8ef73f84ababd62df5524\": container with ID starting with b4a5b355356433077dd8d61ec7dec848b459dadbaaf8ef73f84ababd62df5524 not found: ID does not exist" containerID="b4a5b355356433077dd8d61ec7dec848b459dadbaaf8ef73f84ababd62df5524"
Sep 10 13:05:59.023767 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.023737    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a5b355356433077dd8d61ec7dec848b459dadbaaf8ef73f84ababd62df5524"} err="failed to get container status \"b4a5b355356433077dd8d61ec7dec848b459dadbaaf8ef73f84ababd62df5524\": rpc error: code = NotFound desc = could not find container \"b4a5b355356433077dd8d61ec7dec848b459dadbaaf8ef73f84ababd62df5524\": container with ID starting with b4a5b355356433077dd8d61ec7dec848b459dadbaaf8ef73f84ababd62df5524 not found: ID does not exist"
Sep 10 13:05:59.030414 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.030395    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"]
Sep 10 13:05:59.035497 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.035475    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk"]
Sep 10 13:05:59.518670 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.518632    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 13:05:59.518851 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:05:59.518707    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-c8b83b-openshift-default-594fdc47d8-ljdhk" podUID="7be809a2-426b-402f-95f0-068f302a38be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 13:06:00.708052 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:00.708009    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7be809a2-426b-402f-95f0-068f302a38be" path="/var/lib/kubelet/pods/7be809a2-426b-402f-95f0-068f302a38be/volumes"
Sep 10 13:06:10.421809 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:10.421771    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:06:18.507151 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:18.507114    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"]
Sep 10 13:06:23.721356 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:23.721318    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:06:27.029174 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:27.029139    2563 scope.go:117] "RemoveContainer" containerID="b2e0288c58aa084fd3baec05ca7040b436fbc31351502db4706cd368ecc1d261"
Sep 10 13:06:33.526819 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.526781    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd" podUID="f1816e32-03a9-47b2-8510-b4089f8ae2e3" containerName="main" containerID="cri-o://fd7008b829654f3bbcfa9458d8490a7f7f06de584740b41b74d9342777c3a282" gracePeriod=45
Sep 10 13:06:33.787314 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.787248    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:06:33.890696 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.890662    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-model-cache\") pod \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") "
Sep 10 13:06:33.890696 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.890702    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-dshm\") pod \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") "
Sep 10 13:06:33.890914 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.890763    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-tmp-dir\") pod \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") "
Sep 10 13:06:33.890914 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.890837    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1816e32-03a9-47b2-8510-b4089f8ae2e3-tls-certs\") pod \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") "
Sep 10 13:06:33.890914 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.890895    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wflsv\" (UniqueName: \"kubernetes.io/projected/f1816e32-03a9-47b2-8510-b4089f8ae2e3-kube-api-access-wflsv\") pod \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") "
Sep 10 13:06:33.891083 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.890920    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-home\") pod \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\" (UID: \"f1816e32-03a9-47b2-8510-b4089f8ae2e3\") "
Sep 10 13:06:33.891083 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.890984    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f1816e32-03a9-47b2-8510-b4089f8ae2e3" (UID: "f1816e32-03a9-47b2-8510-b4089f8ae2e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:06:33.891083 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.891069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f1816e32-03a9-47b2-8510-b4089f8ae2e3" (UID: "f1816e32-03a9-47b2-8510-b4089f8ae2e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:06:33.891308 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.891268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-home" (OuterVolumeSpecName: "home") pod "f1816e32-03a9-47b2-8510-b4089f8ae2e3" (UID: "f1816e32-03a9-47b2-8510-b4089f8ae2e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:06:33.891429 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.891327    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:06:33.891429 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.891348    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:06:33.893259 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.893236    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-dshm" (OuterVolumeSpecName: "dshm") pod "f1816e32-03a9-47b2-8510-b4089f8ae2e3" (UID: "f1816e32-03a9-47b2-8510-b4089f8ae2e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:06:33.893427 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.893410    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1816e32-03a9-47b2-8510-b4089f8ae2e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1816e32-03a9-47b2-8510-b4089f8ae2e3" (UID: "f1816e32-03a9-47b2-8510-b4089f8ae2e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:06:33.893491 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.893448    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1816e32-03a9-47b2-8510-b4089f8ae2e3-kube-api-access-wflsv" (OuterVolumeSpecName: "kube-api-access-wflsv") pod "f1816e32-03a9-47b2-8510-b4089f8ae2e3" (UID: "f1816e32-03a9-47b2-8510-b4089f8ae2e3"). InnerVolumeSpecName "kube-api-access-wflsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:06:33.992337 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.992278    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wflsv\" (UniqueName: \"kubernetes.io/projected/f1816e32-03a9-47b2-8510-b4089f8ae2e3-kube-api-access-wflsv\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:06:33.992337 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.992324    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:06:33.992337 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.992336    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1816e32-03a9-47b2-8510-b4089f8ae2e3-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:06:33.992337 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:33.992344    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1816e32-03a9-47b2-8510-b4089f8ae2e3-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:06:34.209379 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:34.209340    2563 generic.go:356] "Generic (PLEG): container finished" podID="f1816e32-03a9-47b2-8510-b4089f8ae2e3" containerID="fd7008b829654f3bbcfa9458d8490a7f7f06de584740b41b74d9342777c3a282" exitCode=0
Sep 10 13:06:34.209529 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:34.209383    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"
Sep 10 13:06:34.209529 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:34.209434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd" event={"ID":"f1816e32-03a9-47b2-8510-b4089f8ae2e3","Type":"ContainerDied","Data":"fd7008b829654f3bbcfa9458d8490a7f7f06de584740b41b74d9342777c3a282"}
Sep 10 13:06:34.209529 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:34.209474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd" event={"ID":"f1816e32-03a9-47b2-8510-b4089f8ae2e3","Type":"ContainerDied","Data":"2c0ed1186d198931f8c6d6220c15b815b00723421f41480020a4f7c37c5d3e43"}
Sep 10 13:06:34.209529 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:34.209490    2563 scope.go:117] "RemoveContainer" containerID="fd7008b829654f3bbcfa9458d8490a7f7f06de584740b41b74d9342777c3a282"
Sep 10 13:06:34.218854 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:34.218832    2563 scope.go:117] "RemoveContainer" containerID="fd7008b829654f3bbcfa9458d8490a7f7f06de584740b41b74d9342777c3a282"
Sep 10 13:06:34.219133 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:06:34.219114    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7008b829654f3bbcfa9458d8490a7f7f06de584740b41b74d9342777c3a282\": container with ID starting with fd7008b829654f3bbcfa9458d8490a7f7f06de584740b41b74d9342777c3a282 not found: ID does not exist" containerID="fd7008b829654f3bbcfa9458d8490a7f7f06de584740b41b74d9342777c3a282"
Sep 10 13:06:34.219179 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:34.219143    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7008b829654f3bbcfa9458d8490a7f7f06de584740b41b74d9342777c3a282"} err="failed to get container status \"fd7008b829654f3bbcfa9458d8490a7f7f06de584740b41b74d9342777c3a282\": rpc error: code = NotFound desc = could not find container \"fd7008b829654f3bbcfa9458d8490a7f7f06de584740b41b74d9342777c3a282\": container with ID starting with fd7008b829654f3bbcfa9458d8490a7f7f06de584740b41b74d9342777c3a282 not found: ID does not exist"
Sep 10 13:06:34.235829 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:34.235795    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"]
Sep 10 13:06:34.242307 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:34.242280    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c91a45/ipp-route-c91a45-kserve-54765dcddc-w7wpd"]
Sep 10 13:06:34.705882 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:06:34.705842    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1816e32-03a9-47b2-8510-b4089f8ae2e3" path="/var/lib/kubelet/pods/f1816e32-03a9-47b2-8510-b4089f8ae2e3/volumes"
Sep 10 13:07:31.642334 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:31.642301    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:07:36.213225 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:36.213192    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"]
Sep 10 13:07:36.213687 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:36.213415    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl" podUID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerName="istio-proxy" containerID="cri-o://b6e13fc4f6c149e2695029ed59a1253ee9de728d2068d3bd92687977d86b6c8e" gracePeriod=30
Sep 10 13:07:41.471480 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.471455    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:07:41.540681 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.540606    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-envoy\") pod \"8de8aa95-a78e-485d-96cd-07b14b62a528\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") "
Sep 10 13:07:41.540681 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.540643    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-workload-socket\") pod \"8de8aa95-a78e-485d-96cd-07b14b62a528\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") "
Sep 10 13:07:41.540681 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.540675    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-workload-certs\") pod \"8de8aa95-a78e-485d-96cd-07b14b62a528\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") "
Sep 10 13:07:41.540960 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.540706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de8aa95-a78e-485d-96cd-07b14b62a528-istiod-ca-cert\") pod \"8de8aa95-a78e-485d-96cd-07b14b62a528\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") "
Sep 10 13:07:41.540960 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.540729    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-token\") pod \"8de8aa95-a78e-485d-96cd-07b14b62a528\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") "
Sep 10 13:07:41.540960 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.540743    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-data\") pod \"8de8aa95-a78e-485d-96cd-07b14b62a528\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") "
Sep 10 13:07:41.540960 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.540833    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-credential-socket\") pod \"8de8aa95-a78e-485d-96cd-07b14b62a528\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") "
Sep 10 13:07:41.540960 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.540877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-podinfo\") pod \"8de8aa95-a78e-485d-96cd-07b14b62a528\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") "
Sep 10 13:07:41.540960 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.540942    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h6c9\" (UniqueName: \"kubernetes.io/projected/8de8aa95-a78e-485d-96cd-07b14b62a528-kube-api-access-2h6c9\") pod \"8de8aa95-a78e-485d-96cd-07b14b62a528\" (UID: \"8de8aa95-a78e-485d-96cd-07b14b62a528\") "
Sep 10 13:07:41.541255 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.541056    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8de8aa95-a78e-485d-96cd-07b14b62a528" (UID: "8de8aa95-a78e-485d-96cd-07b14b62a528"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:07:41.541255 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.541087    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-data" (OuterVolumeSpecName: "istio-data") pod "8de8aa95-a78e-485d-96cd-07b14b62a528" (UID: "8de8aa95-a78e-485d-96cd-07b14b62a528"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:07:41.541255 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.541145    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8de8aa95-a78e-485d-96cd-07b14b62a528-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8de8aa95-a78e-485d-96cd-07b14b62a528" (UID: "8de8aa95-a78e-485d-96cd-07b14b62a528"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:07:41.541399 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.541279    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8de8aa95-a78e-485d-96cd-07b14b62a528" (UID: "8de8aa95-a78e-485d-96cd-07b14b62a528"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:07:41.541399 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.541378    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8de8aa95-a78e-485d-96cd-07b14b62a528" (UID: "8de8aa95-a78e-485d-96cd-07b14b62a528"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:07:41.541399 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.541394    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-workload-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:07:41.541593 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.541413    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8de8aa95-a78e-485d-96cd-07b14b62a528-istiod-ca-cert\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:07:41.541593 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.541426    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-data\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:07:41.541593 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.541439    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-credential-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:07:41.542951 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.542926    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8de8aa95-a78e-485d-96cd-07b14b62a528" (UID: "8de8aa95-a78e-485d-96cd-07b14b62a528"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:07:41.543074 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.542971    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-token" (OuterVolumeSpecName: "istio-token") pod "8de8aa95-a78e-485d-96cd-07b14b62a528" (UID: "8de8aa95-a78e-485d-96cd-07b14b62a528"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:07:41.543074 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.543019    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de8aa95-a78e-485d-96cd-07b14b62a528-kube-api-access-2h6c9" (OuterVolumeSpecName: "kube-api-access-2h6c9") pod "8de8aa95-a78e-485d-96cd-07b14b62a528" (UID: "8de8aa95-a78e-485d-96cd-07b14b62a528"). InnerVolumeSpecName "kube-api-access-2h6c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:07:41.543193 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.543171    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8de8aa95-a78e-485d-96cd-07b14b62a528" (UID: "8de8aa95-a78e-485d-96cd-07b14b62a528"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:07:41.595576 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.595520    2563 generic.go:356] "Generic (PLEG): container finished" podID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerID="b6e13fc4f6c149e2695029ed59a1253ee9de728d2068d3bd92687977d86b6c8e" exitCode=0
Sep 10 13:07:41.595735 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.595588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl" event={"ID":"8de8aa95-a78e-485d-96cd-07b14b62a528","Type":"ContainerDied","Data":"b6e13fc4f6c149e2695029ed59a1253ee9de728d2068d3bd92687977d86b6c8e"}
Sep 10 13:07:41.595735 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.595600    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"
Sep 10 13:07:41.595735 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.595689    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl" event={"ID":"8de8aa95-a78e-485d-96cd-07b14b62a528","Type":"ContainerDied","Data":"ff1bd861b667bbd9b4d76f2a6c6e0482d6af1002914228d9a5c497c03a3bf4d1"}
Sep 10 13:07:41.595735 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.595723    2563 scope.go:117] "RemoveContainer" containerID="b6e13fc4f6c149e2695029ed59a1253ee9de728d2068d3bd92687977d86b6c8e"
Sep 10 13:07:41.605615 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.605597    2563 scope.go:117] "RemoveContainer" containerID="74a49fcd1c9367d074a153ee8e22031a22326b4165a9fc685b895c3340aeba8a"
Sep 10 13:07:41.613595 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.613537    2563 scope.go:117] "RemoveContainer" containerID="b6e13fc4f6c149e2695029ed59a1253ee9de728d2068d3bd92687977d86b6c8e"
Sep 10 13:07:41.613856 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:07:41.613834    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e13fc4f6c149e2695029ed59a1253ee9de728d2068d3bd92687977d86b6c8e\": container with ID starting with b6e13fc4f6c149e2695029ed59a1253ee9de728d2068d3bd92687977d86b6c8e not found: ID does not exist" containerID="b6e13fc4f6c149e2695029ed59a1253ee9de728d2068d3bd92687977d86b6c8e"
Sep 10 13:07:41.613925 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.613864    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e13fc4f6c149e2695029ed59a1253ee9de728d2068d3bd92687977d86b6c8e"} err="failed to get container status \"b6e13fc4f6c149e2695029ed59a1253ee9de728d2068d3bd92687977d86b6c8e\": rpc error: code = NotFound desc = could not find container \"b6e13fc4f6c149e2695029ed59a1253ee9de728d2068d3bd92687977d86b6c8e\": container with ID starting with b6e13fc4f6c149e2695029ed59a1253ee9de728d2068d3bd92687977d86b6c8e not found: ID does not exist"
Sep 10 13:07:41.613925 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.613881    2563 scope.go:117] "RemoveContainer" containerID="74a49fcd1c9367d074a153ee8e22031a22326b4165a9fc685b895c3340aeba8a"
Sep 10 13:07:41.614133 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:07:41.614118    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a49fcd1c9367d074a153ee8e22031a22326b4165a9fc685b895c3340aeba8a\": container with ID starting with 74a49fcd1c9367d074a153ee8e22031a22326b4165a9fc685b895c3340aeba8a not found: ID does not exist" containerID="74a49fcd1c9367d074a153ee8e22031a22326b4165a9fc685b895c3340aeba8a"
Sep 10 13:07:41.614173 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.614137    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a49fcd1c9367d074a153ee8e22031a22326b4165a9fc685b895c3340aeba8a"} err="failed to get container status \"74a49fcd1c9367d074a153ee8e22031a22326b4165a9fc685b895c3340aeba8a\": rpc error: code = NotFound desc = could not find container \"74a49fcd1c9367d074a153ee8e22031a22326b4165a9fc685b895c3340aeba8a\": container with ID starting with 74a49fcd1c9367d074a153ee8e22031a22326b4165a9fc685b895c3340aeba8a not found: ID does not exist"
Sep 10 13:07:41.642195 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.642173    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-token\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:07:41.642195 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.642196    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-podinfo\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:07:41.642329 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.642206    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h6c9\" (UniqueName: \"kubernetes.io/projected/8de8aa95-a78e-485d-96cd-07b14b62a528-kube-api-access-2h6c9\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:07:41.642329 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.642216    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-istio-envoy\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:07:41.642329 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.642224    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8de8aa95-a78e-485d-96cd-07b14b62a528-workload-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:07:41.643118 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.643098    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"]
Sep 10 13:07:41.657698 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:41.657677    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-6ea780-openshift-default-5cc54b94b-26xjl"]
Sep 10 13:07:42.705791 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:42.705756    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8de8aa95-a78e-485d-96cd-07b14b62a528" path="/var/lib/kubelet/pods/8de8aa95-a78e-485d-96cd-07b14b62a528/volumes"
Sep 10 13:07:50.652260 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:07:50.652176    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:08:07.459261 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.458706    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"]
Sep 10 13:08:07.459261 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.459014    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9" podUID="70392a30-b812-4ec2-b7ca-c8fac767edc7" containerName="maas-api" containerID="cri-o://beed0f69f2a7ce4d8545c78fed4507c5311a056a1f005ad4499cb3cbe0b1c01e" gracePeriod=30
Sep 10 13:08:07.720038 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.719971    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 13:08:07.739967 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.739935    2563 generic.go:356] "Generic (PLEG): container finished" podID="70392a30-b812-4ec2-b7ca-c8fac767edc7" containerID="beed0f69f2a7ce4d8545c78fed4507c5311a056a1f005ad4499cb3cbe0b1c01e" exitCode=0
Sep 10 13:08:07.740175 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.739981    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"
Sep 10 13:08:07.740175 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.740028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9" event={"ID":"70392a30-b812-4ec2-b7ca-c8fac767edc7","Type":"ContainerDied","Data":"beed0f69f2a7ce4d8545c78fed4507c5311a056a1f005ad4499cb3cbe0b1c01e"}
Sep 10 13:08:07.740175 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.740094    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9" event={"ID":"70392a30-b812-4ec2-b7ca-c8fac767edc7","Type":"ContainerDied","Data":"7faddb10424bfd443978284a70f2f3b4d5f6085e4aadd246c67fbe9c42f5d66f"}
Sep 10 13:08:07.740175 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.740126    2563 scope.go:117] "RemoveContainer" containerID="beed0f69f2a7ce4d8545c78fed4507c5311a056a1f005ad4499cb3cbe0b1c01e"
Sep 10 13:08:07.753097 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.752970    2563 scope.go:117] "RemoveContainer" containerID="beed0f69f2a7ce4d8545c78fed4507c5311a056a1f005ad4499cb3cbe0b1c01e"
Sep 10 13:08:07.753533 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:08:07.753509    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beed0f69f2a7ce4d8545c78fed4507c5311a056a1f005ad4499cb3cbe0b1c01e\": container with ID starting with beed0f69f2a7ce4d8545c78fed4507c5311a056a1f005ad4499cb3cbe0b1c01e not found: ID does not exist" containerID="beed0f69f2a7ce4d8545c78fed4507c5311a056a1f005ad4499cb3cbe0b1c01e"
Sep 10 13:08:07.753758 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.753543    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beed0f69f2a7ce4d8545c78fed4507c5311a056a1f005ad4499cb3cbe0b1c01e"} err="failed to get container status \"beed0f69f2a7ce4d8545c78fed4507c5311a056a1f005ad4499cb3cbe0b1c01e\": rpc error: code = NotFound desc = could not find container \"beed0f69f2a7ce4d8545c78fed4507c5311a056a1f005ad4499cb3cbe0b1c01e\": container with ID starting with beed0f69f2a7ce4d8545c78fed4507c5311a056a1f005ad4499cb3cbe0b1c01e not found: ID does not exist"
Sep 10 13:08:07.800093 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.800052    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxw2b\" (UniqueName: \"kubernetes.io/projected/70392a30-b812-4ec2-b7ca-c8fac767edc7-kube-api-access-vxw2b\") pod \"70392a30-b812-4ec2-b7ca-c8fac767edc7\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") "
Sep 10 13:08:07.800270 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.800121    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-maas-api-tls\") pod \"70392a30-b812-4ec2-b7ca-c8fac767edc7\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") "
Sep 10 13:08:07.800270 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.800185    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-metrics-certs\") pod \"70392a30-b812-4ec2-b7ca-c8fac767edc7\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") "
Sep 10 13:08:07.800352 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.800272    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70392a30-b812-4ec2-b7ca-c8fac767edc7-openshift-service-ca\") pod \"70392a30-b812-4ec2-b7ca-c8fac767edc7\" (UID: \"70392a30-b812-4ec2-b7ca-c8fac767edc7\") "
Sep 10 13:08:07.800702 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.800675    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70392a30-b812-4ec2-b7ca-c8fac767edc7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70392a30-b812-4ec2-b7ca-c8fac767edc7" (UID: "70392a30-b812-4ec2-b7ca-c8fac767edc7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:08:07.802286 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.802259    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "70392a30-b812-4ec2-b7ca-c8fac767edc7" (UID: "70392a30-b812-4ec2-b7ca-c8fac767edc7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:08:07.802401 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.802314    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70392a30-b812-4ec2-b7ca-c8fac767edc7" (UID: "70392a30-b812-4ec2-b7ca-c8fac767edc7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:08:07.802457 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.802412    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70392a30-b812-4ec2-b7ca-c8fac767edc7-kube-api-access-vxw2b" (OuterVolumeSpecName: "kube-api-access-vxw2b") pod "70392a30-b812-4ec2-b7ca-c8fac767edc7" (UID: "70392a30-b812-4ec2-b7ca-c8fac767edc7"). InnerVolumeSpecName "kube-api-access-vxw2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:08:07.901395 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.901354    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70392a30-b812-4ec2-b7ca-c8fac767edc7-openshift-service-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:08:07.901395 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.901388    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxw2b\" (UniqueName: \"kubernetes.io/projected/70392a30-b812-4ec2-b7ca-c8fac767edc7-kube-api-access-vxw2b\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:08:07.901395 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.901398    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-maas-api-tls\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:08:07.901675 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:07.901407    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70392a30-b812-4ec2-b7ca-c8fac767edc7-metrics-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:08:08.067795 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:08.067763    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"]
Sep 10 13:08:08.082044 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:08.082010    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e5208b-747686f9bb-ssrs9"]
Sep 10 13:08:08.704450 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:08.704412    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70392a30-b812-4ec2-b7ca-c8fac767edc7" path="/var/lib/kubelet/pods/70392a30-b812-4ec2-b7ca-c8fac767edc7/volumes"
Sep 10 13:08:10.359946 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:10.359906    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:08:15.209083 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:15.209048    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"]
Sep 10 13:08:15.209596 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:15.209272    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerName="istio-proxy" containerID="cri-o://c31c636da0a6ccaef30f46828a0a3eb1e3b48114be3f3f9a6ef66b0740d411d2" gracePeriod=30
Sep 10 13:08:19.323447 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:19.323404    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:08:20.503341 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.503302    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 13:08:20.524736 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.524707    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-token\") pod \"2766220e-dd8b-4305-b025-e1799a9a84e9\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") "
Sep 10 13:08:20.524736 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.524739    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-data\") pod \"2766220e-dd8b-4305-b025-e1799a9a84e9\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") "
Sep 10 13:08:20.524970 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.524773    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-workload-certs\") pod \"2766220e-dd8b-4305-b025-e1799a9a84e9\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") "
Sep 10 13:08:20.524970 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.524805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltsgw\" (UniqueName: \"kubernetes.io/projected/2766220e-dd8b-4305-b025-e1799a9a84e9-kube-api-access-ltsgw\") pod \"2766220e-dd8b-4305-b025-e1799a9a84e9\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") "
Sep 10 13:08:20.524970 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.524832    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-credential-socket\") pod \"2766220e-dd8b-4305-b025-e1799a9a84e9\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") "
Sep 10 13:08:20.524970 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.524869    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-workload-socket\") pod \"2766220e-dd8b-4305-b025-e1799a9a84e9\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") "
Sep 10 13:08:20.524970 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.524892    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-envoy\") pod \"2766220e-dd8b-4305-b025-e1799a9a84e9\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") "
Sep 10 13:08:20.524970 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.524945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-podinfo\") pod \"2766220e-dd8b-4305-b025-e1799a9a84e9\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") "
Sep 10 13:08:20.525286 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.524977    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2766220e-dd8b-4305-b025-e1799a9a84e9-istiod-ca-cert\") pod \"2766220e-dd8b-4305-b025-e1799a9a84e9\" (UID: \"2766220e-dd8b-4305-b025-e1799a9a84e9\") "
Sep 10 13:08:20.525286 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.525007    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-data" (OuterVolumeSpecName: "istio-data") pod "2766220e-dd8b-4305-b025-e1799a9a84e9" (UID: "2766220e-dd8b-4305-b025-e1799a9a84e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:08:20.525286 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.525188    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2766220e-dd8b-4305-b025-e1799a9a84e9" (UID: "2766220e-dd8b-4305-b025-e1799a9a84e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:08:20.525286 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.525228    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2766220e-dd8b-4305-b025-e1799a9a84e9" (UID: "2766220e-dd8b-4305-b025-e1799a9a84e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:08:20.525484 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.525341    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2766220e-dd8b-4305-b025-e1799a9a84e9" (UID: "2766220e-dd8b-4305-b025-e1799a9a84e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:08:20.525484 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.525367    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2766220e-dd8b-4305-b025-e1799a9a84e9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2766220e-dd8b-4305-b025-e1799a9a84e9" (UID: "2766220e-dd8b-4305-b025-e1799a9a84e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:08:20.525612 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.525485    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-credential-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:08:20.525612 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.525504    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-workload-socket\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:08:20.525612 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.525518    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2766220e-dd8b-4305-b025-e1799a9a84e9-istiod-ca-cert\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:08:20.525612 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.525534    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-data\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:08:20.525612 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.525564    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-workload-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:08:20.527185 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.527151    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-token" (OuterVolumeSpecName: "istio-token") pod "2766220e-dd8b-4305-b025-e1799a9a84e9" (UID: "2766220e-dd8b-4305-b025-e1799a9a84e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:08:20.527185 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.527163    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2766220e-dd8b-4305-b025-e1799a9a84e9" (UID: "2766220e-dd8b-4305-b025-e1799a9a84e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 13:08:20.527185 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.527166    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2766220e-dd8b-4305-b025-e1799a9a84e9-kube-api-access-ltsgw" (OuterVolumeSpecName: "kube-api-access-ltsgw") pod "2766220e-dd8b-4305-b025-e1799a9a84e9" (UID: "2766220e-dd8b-4305-b025-e1799a9a84e9"). InnerVolumeSpecName "kube-api-access-ltsgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:08:20.527365 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.527168    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2766220e-dd8b-4305-b025-e1799a9a84e9" (UID: "2766220e-dd8b-4305-b025-e1799a9a84e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:08:20.626347 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.626312    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-token\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:08:20.626347 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.626345    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltsgw\" (UniqueName: \"kubernetes.io/projected/2766220e-dd8b-4305-b025-e1799a9a84e9-kube-api-access-ltsgw\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:08:20.626347 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.626355    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-envoy\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:08:20.626627 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.626363    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2766220e-dd8b-4305-b025-e1799a9a84e9-istio-podinfo\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:08:20.813824 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.813726    2563 generic.go:356] "Generic (PLEG): container finished" podID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerID="c31c636da0a6ccaef30f46828a0a3eb1e3b48114be3f3f9a6ef66b0740d411d2" exitCode=0
Sep 10 13:08:20.813824 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.813787    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"
Sep 10 13:08:20.813824 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.813799    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" event={"ID":"2766220e-dd8b-4305-b025-e1799a9a84e9","Type":"ContainerDied","Data":"c31c636da0a6ccaef30f46828a0a3eb1e3b48114be3f3f9a6ef66b0740d411d2"}
Sep 10 13:08:20.814074 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.813876    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl" event={"ID":"2766220e-dd8b-4305-b025-e1799a9a84e9","Type":"ContainerDied","Data":"aecf850c105a9a0bd5f64aad46027c69a6fd23c4f5a35aacb25e8101519ceb7f"}
Sep 10 13:08:20.814074 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.813897    2563 scope.go:117] "RemoveContainer" containerID="c31c636da0a6ccaef30f46828a0a3eb1e3b48114be3f3f9a6ef66b0740d411d2"
Sep 10 13:08:20.822709 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.822691    2563 scope.go:117] "RemoveContainer" containerID="bb5196a793fba35be4a009b2a5120a7bc6a4ce77215bd6182a3d833fd2da17e8"
Sep 10 13:08:20.830927 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.830912    2563 scope.go:117] "RemoveContainer" containerID="c31c636da0a6ccaef30f46828a0a3eb1e3b48114be3f3f9a6ef66b0740d411d2"
Sep 10 13:08:20.831207 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:08:20.831189    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31c636da0a6ccaef30f46828a0a3eb1e3b48114be3f3f9a6ef66b0740d411d2\": container with ID starting with c31c636da0a6ccaef30f46828a0a3eb1e3b48114be3f3f9a6ef66b0740d411d2 not found: ID does not exist" containerID="c31c636da0a6ccaef30f46828a0a3eb1e3b48114be3f3f9a6ef66b0740d411d2"
Sep 10 13:08:20.831265 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.831215    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31c636da0a6ccaef30f46828a0a3eb1e3b48114be3f3f9a6ef66b0740d411d2"} err="failed to get container status \"c31c636da0a6ccaef30f46828a0a3eb1e3b48114be3f3f9a6ef66b0740d411d2\": rpc error: code = NotFound desc = could not find container \"c31c636da0a6ccaef30f46828a0a3eb1e3b48114be3f3f9a6ef66b0740d411d2\": container with ID starting with c31c636da0a6ccaef30f46828a0a3eb1e3b48114be3f3f9a6ef66b0740d411d2 not found: ID does not exist"
Sep 10 13:08:20.831265 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.831230    2563 scope.go:117] "RemoveContainer" containerID="bb5196a793fba35be4a009b2a5120a7bc6a4ce77215bd6182a3d833fd2da17e8"
Sep 10 13:08:20.831479 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:08:20.831461    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5196a793fba35be4a009b2a5120a7bc6a4ce77215bd6182a3d833fd2da17e8\": container with ID starting with bb5196a793fba35be4a009b2a5120a7bc6a4ce77215bd6182a3d833fd2da17e8 not found: ID does not exist" containerID="bb5196a793fba35be4a009b2a5120a7bc6a4ce77215bd6182a3d833fd2da17e8"
Sep 10 13:08:20.831535 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.831489    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5196a793fba35be4a009b2a5120a7bc6a4ce77215bd6182a3d833fd2da17e8"} err="failed to get container status \"bb5196a793fba35be4a009b2a5120a7bc6a4ce77215bd6182a3d833fd2da17e8\": rpc error: code = NotFound desc = could not find container \"bb5196a793fba35be4a009b2a5120a7bc6a4ce77215bd6182a3d833fd2da17e8\": container with ID starting with bb5196a793fba35be4a009b2a5120a7bc6a4ce77215bd6182a3d833fd2da17e8 not found: ID does not exist"
Sep 10 13:08:20.835680 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.835658    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"]
Sep 10 13:08:20.843189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:20.843169    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e5208b-openshift-default-84cccb665d-s9vwl"]
Sep 10 13:08:22.705231 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:08:22.705197    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" path="/var/lib/kubelet/pods/2766220e-dd8b-4305-b025-e1799a9a84e9/volumes"
Sep 10 13:09:43.424683 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:09:43.424645    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:10:09.760105 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:09.760067    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:10:18.319261 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:18.319222    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:10:27.639854 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:27.639811    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:10:44.225514 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:44.225477    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:10:51.162817 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:51.162735    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:10:54.395038 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:54.394998    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"]
Sep 10 13:10:54.395495 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:54.395203    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t" podUID="ce40549a-16c6-4438-995e-01f1f68546da" containerName="manager" containerID="cri-o://266d5f51ff1f2ccdf069998780dfa4db299f603502074c20558672f1da1b6f6d" gracePeriod=10
Sep 10 13:10:54.957596 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:54.957569    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 13:10:55.029163 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.029066    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-249d5\" (UniqueName: \"kubernetes.io/projected/ce40549a-16c6-4438-995e-01f1f68546da-kube-api-access-249d5\") pod \"ce40549a-16c6-4438-995e-01f1f68546da\" (UID: \"ce40549a-16c6-4438-995e-01f1f68546da\") "
Sep 10 13:10:55.029335 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.029274    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce40549a-16c6-4438-995e-01f1f68546da-extensions-socket-volume\") pod \"ce40549a-16c6-4438-995e-01f1f68546da\" (UID: \"ce40549a-16c6-4438-995e-01f1f68546da\") "
Sep 10 13:10:55.029712 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.029684    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce40549a-16c6-4438-995e-01f1f68546da-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ce40549a-16c6-4438-995e-01f1f68546da" (UID: "ce40549a-16c6-4438-995e-01f1f68546da"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:10:55.031311 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.031279    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce40549a-16c6-4438-995e-01f1f68546da-kube-api-access-249d5" (OuterVolumeSpecName: "kube-api-access-249d5") pod "ce40549a-16c6-4438-995e-01f1f68546da" (UID: "ce40549a-16c6-4438-995e-01f1f68546da"). InnerVolumeSpecName "kube-api-access-249d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:10:55.130901 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.130862    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-249d5\" (UniqueName: \"kubernetes.io/projected/ce40549a-16c6-4438-995e-01f1f68546da-kube-api-access-249d5\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:10:55.130901 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.130891    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce40549a-16c6-4438-995e-01f1f68546da-extensions-socket-volume\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:10:55.632115 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.632082    2563 generic.go:356] "Generic (PLEG): container finished" podID="ce40549a-16c6-4438-995e-01f1f68546da" containerID="266d5f51ff1f2ccdf069998780dfa4db299f603502074c20558672f1da1b6f6d" exitCode=0
Sep 10 13:10:55.632599 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.632130    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"
Sep 10 13:10:55.632599 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.632166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t" event={"ID":"ce40549a-16c6-4438-995e-01f1f68546da","Type":"ContainerDied","Data":"266d5f51ff1f2ccdf069998780dfa4db299f603502074c20558672f1da1b6f6d"}
Sep 10 13:10:55.632599 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.632221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t" event={"ID":"ce40549a-16c6-4438-995e-01f1f68546da","Type":"ContainerDied","Data":"34cf904f619fad27da9b573054abf493b568f97fed578941f33234ab4a32536c"}
Sep 10 13:10:55.632599 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.632243    2563 scope.go:117] "RemoveContainer" containerID="266d5f51ff1f2ccdf069998780dfa4db299f603502074c20558672f1da1b6f6d"
Sep 10 13:10:55.641688 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.641671    2563 scope.go:117] "RemoveContainer" containerID="e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b"
Sep 10 13:10:55.649870 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.649854    2563 scope.go:117] "RemoveContainer" containerID="266d5f51ff1f2ccdf069998780dfa4db299f603502074c20558672f1da1b6f6d"
Sep 10 13:10:55.650154 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:10:55.650135    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"266d5f51ff1f2ccdf069998780dfa4db299f603502074c20558672f1da1b6f6d\": container with ID starting with 266d5f51ff1f2ccdf069998780dfa4db299f603502074c20558672f1da1b6f6d not found: ID does not exist" containerID="266d5f51ff1f2ccdf069998780dfa4db299f603502074c20558672f1da1b6f6d"
Sep 10 13:10:55.650200 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.650166    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"266d5f51ff1f2ccdf069998780dfa4db299f603502074c20558672f1da1b6f6d"} err="failed to get container status \"266d5f51ff1f2ccdf069998780dfa4db299f603502074c20558672f1da1b6f6d\": rpc error: code = NotFound desc = could not find container \"266d5f51ff1f2ccdf069998780dfa4db299f603502074c20558672f1da1b6f6d\": container with ID starting with 266d5f51ff1f2ccdf069998780dfa4db299f603502074c20558672f1da1b6f6d not found: ID does not exist"
Sep 10 13:10:55.650200 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.650185    2563 scope.go:117] "RemoveContainer" containerID="e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b"
Sep 10 13:10:55.650448 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:10:55.650431    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b\": container with ID starting with e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b not found: ID does not exist" containerID="e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b"
Sep 10 13:10:55.650485 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.650456    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b"} err="failed to get container status \"e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b\": rpc error: code = NotFound desc = could not find container \"e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b\": container with ID starting with e2b2d1a46f4a90d21f95debd43797a56c449627598d82997bdde00cb1ffd701b not found: ID does not exist"
Sep 10 13:10:55.659324 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.659301    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"]
Sep 10 13:10:55.670180 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:55.670141    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vjw4t"]
Sep 10 13:10:56.704490 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:10:56.704451    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce40549a-16c6-4438-995e-01f1f68546da" path="/var/lib/kubelet/pods/ce40549a-16c6-4438-995e-01f1f68546da/volumes"
Sep 10 13:11:55.311308 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311274    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5"]
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311688    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerName="istio-proxy"
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311702    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerName="istio-proxy"
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311710    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerName="istio-proxy"
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311716    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerName="istio-proxy"
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311725    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerName="istio-proxy"
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311731    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerName="istio-proxy"
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311741    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerName="istio-proxy"
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311746    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerName="istio-proxy"
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311751    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1816e32-03a9-47b2-8510-b4089f8ae2e3" containerName="main"
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311756    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1816e32-03a9-47b2-8510-b4089f8ae2e3" containerName="main"
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311763    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce40549a-16c6-4438-995e-01f1f68546da" containerName="manager"
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311767    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce40549a-16c6-4438-995e-01f1f68546da" containerName="manager"
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311776    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7be809a2-426b-402f-95f0-068f302a38be" containerName="istio-proxy"
Sep 10 13:11:55.311778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311781    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7be809a2-426b-402f-95f0-068f302a38be" containerName="istio-proxy"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311788    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70392a30-b812-4ec2-b7ca-c8fac767edc7" containerName="maas-api"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311792    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="70392a30-b812-4ec2-b7ca-c8fac767edc7" containerName="maas-api"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311800    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce40549a-16c6-4438-995e-01f1f68546da" containerName="manager"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311805    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce40549a-16c6-4438-995e-01f1f68546da" containerName="manager"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311818    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce40549a-16c6-4438-995e-01f1f68546da" containerName="manager"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311823    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce40549a-16c6-4438-995e-01f1f68546da" containerName="manager"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311877    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1816e32-03a9-47b2-8510-b4089f8ae2e3" containerName="main"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311883    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7be809a2-426b-402f-95f0-068f302a38be" containerName="istio-proxy"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311889    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce40549a-16c6-4438-995e-01f1f68546da" containerName="manager"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311894    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerName="istio-proxy"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311900    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerName="istio-proxy"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311906    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce40549a-16c6-4438-995e-01f1f68546da" containerName="manager"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311911    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de8aa95-a78e-485d-96cd-07b14b62a528" containerName="istio-proxy"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311916    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2766220e-dd8b-4305-b025-e1799a9a84e9" containerName="istio-proxy"
Sep 10 13:11:55.312189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.311922    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="70392a30-b812-4ec2-b7ca-c8fac767edc7" containerName="maas-api"
Sep 10 13:11:55.314958 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.314943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5"
Sep 10 13:11:55.320288 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.320268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v8mmc\""
Sep 10 13:11:55.359792 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.359765    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5"]
Sep 10 13:11:55.492630 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.492596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9cefaa6-82f9-4852-86f8-7eda40fb06d6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hd9d5\" (UID: \"e9cefaa6-82f9-4852-86f8-7eda40fb06d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5"
Sep 10 13:11:55.492796 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.492674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6h5j\" (UniqueName: \"kubernetes.io/projected/e9cefaa6-82f9-4852-86f8-7eda40fb06d6-kube-api-access-v6h5j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hd9d5\" (UID: \"e9cefaa6-82f9-4852-86f8-7eda40fb06d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5"
Sep 10 13:11:55.593854 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.593768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6h5j\" (UniqueName: \"kubernetes.io/projected/e9cefaa6-82f9-4852-86f8-7eda40fb06d6-kube-api-access-v6h5j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hd9d5\" (UID: \"e9cefaa6-82f9-4852-86f8-7eda40fb06d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5"
Sep 10 13:11:55.594000 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.593873    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9cefaa6-82f9-4852-86f8-7eda40fb06d6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hd9d5\" (UID: \"e9cefaa6-82f9-4852-86f8-7eda40fb06d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5"
Sep 10 13:11:55.594296 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.594277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e9cefaa6-82f9-4852-86f8-7eda40fb06d6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hd9d5\" (UID: \"e9cefaa6-82f9-4852-86f8-7eda40fb06d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5"
Sep 10 13:11:55.606186 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.606149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6h5j\" (UniqueName: \"kubernetes.io/projected/e9cefaa6-82f9-4852-86f8-7eda40fb06d6-kube-api-access-v6h5j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hd9d5\" (UID: \"e9cefaa6-82f9-4852-86f8-7eda40fb06d6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5"
Sep 10 13:11:55.624963 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.624921    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5"
Sep 10 13:11:55.943488 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:55.943464    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5"]
Sep 10 13:11:55.946334 ip-10-0-133-150 kubenswrapper[2563]: W0910 13:11:55.946309    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cefaa6_82f9_4852_86f8_7eda40fb06d6.slice/crio-30ee5e0c02c41e73885b52983e65979cca3f3cfdc94d7fca01c01c65c8b659f3 WatchSource:0}: Error finding container 30ee5e0c02c41e73885b52983e65979cca3f3cfdc94d7fca01c01c65c8b659f3: Status 404 returned error can't find the container with id 30ee5e0c02c41e73885b52983e65979cca3f3cfdc94d7fca01c01c65c8b659f3
Sep 10 13:11:56.951820 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:56.951787    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5" event={"ID":"e9cefaa6-82f9-4852-86f8-7eda40fb06d6","Type":"ContainerStarted","Data":"69b340d947193d3fb4cd82217fa7ef83e4e79b206c919852ca5ae4b5b09f8f2c"}
Sep 10 13:11:56.952265 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:56.952236    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5"
Sep 10 13:11:56.952328 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:56.952286    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5" event={"ID":"e9cefaa6-82f9-4852-86f8-7eda40fb06d6","Type":"ContainerStarted","Data":"30ee5e0c02c41e73885b52983e65979cca3f3cfdc94d7fca01c01c65c8b659f3"}
Sep 10 13:11:56.996151 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:11:56.996096    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5" podStartSLOduration=1.9960793319999999 podStartE2EDuration="1.996079332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:11:56.993156047 +0000 UTC m=+2490.918067535" watchObservedRunningTime="2026-09-10 13:11:56.996079332 +0000 UTC m=+2490.920990820"
Sep 10 13:12:07.958664 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:07.958628    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hd9d5"
Sep 10 13:12:26.462362 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:26.462268    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:12:35.227174 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:35.227133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:12:42.450853 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.450815    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"]
Sep 10 13:12:42.451252 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.451234    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7be809a2-426b-402f-95f0-068f302a38be" containerName="istio-proxy"
Sep 10 13:12:42.451252 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.451247    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7be809a2-426b-402f-95f0-068f302a38be" containerName="istio-proxy"
Sep 10 13:12:42.451343 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.451332    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7be809a2-426b-402f-95f0-068f302a38be" containerName="istio-proxy"
Sep 10 13:12:42.451380 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.451346    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce40549a-16c6-4438-995e-01f1f68546da" containerName="manager"
Sep 10 13:12:42.454541 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.454520    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.457466 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.457439    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-43506975-kserve-self-signed-certs\""
Sep 10 13:12:42.474685 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.474651    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"]
Sep 10 13:12:42.547733 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.547696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78051822-5469-49e8-b13d-04c823cd9589-tls-certs\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.547733 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.547746    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-model-cache\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.548016 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.547831    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-home\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.548016 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.547859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-tmp-dir\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.548016 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.547960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpdss\" (UniqueName: \"kubernetes.io/projected/78051822-5469-49e8-b13d-04c823cd9589-kube-api-access-lpdss\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.548159 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.548074    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-dshm\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.649608 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.649565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-home\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.649778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.649621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-tmp-dir\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.649778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.649652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpdss\" (UniqueName: \"kubernetes.io/projected/78051822-5469-49e8-b13d-04c823cd9589-kube-api-access-lpdss\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.649778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.649708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-dshm\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.649778 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.649772    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78051822-5469-49e8-b13d-04c823cd9589-tls-certs\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.650275 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.650082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-home\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.650275 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.650088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-model-cache\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.650275 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.650122    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-tmp-dir\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.650496 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.650346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-model-cache\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.652438 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.652415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78051822-5469-49e8-b13d-04c823cd9589-tls-certs\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.652527 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.652474    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-dshm\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.668827 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.668795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpdss\" (UniqueName: \"kubernetes.io/projected/78051822-5469-49e8-b13d-04c823cd9589-kube-api-access-lpdss\") pod \"e2e-same-id-a-43506975-kserve-54b79dd59-49rsp\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") " pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:42.763970 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:42.763873    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:43.052683 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:43.052598    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"]
Sep 10 13:12:43.061635 ip-10-0-133-150 kubenswrapper[2563]: W0910 13:12:43.061600    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78051822_5469_49e8_b13d_04c823cd9589.slice/crio-badc1323aec9feefb6bd9b05d49bb7a06c33e7d4a543c5e2b69bc3023ee4dbf4 WatchSource:0}: Error finding container badc1323aec9feefb6bd9b05d49bb7a06c33e7d4a543c5e2b69bc3023ee4dbf4: Status 404 returned error can't find the container with id badc1323aec9feefb6bd9b05d49bb7a06c33e7d4a543c5e2b69bc3023ee4dbf4
Sep 10 13:12:43.213133 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:43.213099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp" event={"ID":"78051822-5469-49e8-b13d-04c823cd9589","Type":"ContainerStarted","Data":"e309929adc608ea6b63e28ca1da702771856e1b0beace79fea603082f6f64704"}
Sep 10 13:12:43.213133 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:43.213141    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp" event={"ID":"78051822-5469-49e8-b13d-04c823cd9589","Type":"ContainerStarted","Data":"badc1323aec9feefb6bd9b05d49bb7a06c33e7d4a543c5e2b69bc3023ee4dbf4"}
Sep 10 13:12:44.142512 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:44.142453    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp" podStartSLOduration=2.14243153 podStartE2EDuration="2.14243153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:12:43.238053998 +0000 UTC m=+2537.162965497" watchObservedRunningTime="2026-09-10 13:12:44.14243153 +0000 UTC m=+2538.067343021"
Sep 10 13:12:44.143593 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:44.143570    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:12:52.764084 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:52.764039    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:52.764667 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:52.764111    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:52.776774 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:52.776742    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:12:53.285156 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:12:53.285122    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:13:00.927520 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:00.927483    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"]
Sep 10 13:13:00.931131 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:00.931110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:00.935319 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:00.935293    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-43506975-kserve-self-signed-certs\""
Sep 10 13:13:00.951371 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:00.951336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"]
Sep 10 13:13:01.027402 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.027360    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-dshm\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.027673 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.027417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-model-cache\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.027673 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.027525    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brlg\" (UniqueName: \"kubernetes.io/projected/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-kube-api-access-9brlg\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.027673 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.027614    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-home\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.027673 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.027664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-tmp-dir\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.027851 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.027722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-tls-certs\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.128722 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.128680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-tls-certs\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.128936 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.128764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-dshm\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.128936 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.128797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-model-cache\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.128936 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.128824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brlg\" (UniqueName: \"kubernetes.io/projected/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-kube-api-access-9brlg\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.128936 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.128849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-home\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.128936 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.128901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-tmp-dir\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.129376 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.129333    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-home\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.129476 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.129347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-tmp-dir\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.129476 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.129397    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-model-cache\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.131238 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.131214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-dshm\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.131436 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.131418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-tls-certs\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.138993 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.138962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brlg\" (UniqueName: \"kubernetes.io/projected/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-kube-api-access-9brlg\") pod \"e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") " pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.241752 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.241650    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:01.522740 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:01.522643    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"]
Sep 10 13:13:01.525143 ip-10-0-133-150 kubenswrapper[2563]: W0910 13:13:01.525110    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc6c860e_5ccd_4f65_8ce3_82cef1b54b9e.slice/crio-c485ed4a079c41b686f8702d62d6e88ff52a03540e9079b6356f4c8e857bf3d4 WatchSource:0}: Error finding container c485ed4a079c41b686f8702d62d6e88ff52a03540e9079b6356f4c8e857bf3d4: Status 404 returned error can't find the container with id c485ed4a079c41b686f8702d62d6e88ff52a03540e9079b6356f4c8e857bf3d4
Sep 10 13:13:02.333682 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:02.333643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr" event={"ID":"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e","Type":"ContainerStarted","Data":"57687afd2c90f84c05dc9230843310b059e12e62d2d6c978361b607574edfada"}
Sep 10 13:13:02.333682 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:02.333691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr" event={"ID":"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e","Type":"ContainerStarted","Data":"c485ed4a079c41b686f8702d62d6e88ff52a03540e9079b6356f4c8e857bf3d4"}
Sep 10 13:13:03.921625 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:03.921559    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr" podStartSLOduration=3.921528711 podStartE2EDuration="3.921528711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:13:02.356154569 +0000 UTC m=+2556.281066058" watchObservedRunningTime="2026-09-10 13:13:03.921528711 +0000 UTC m=+2557.846440199"
Sep 10 13:13:03.922021 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:03.921871    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:13:11.242769 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:11.242727    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:11.243205 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:11.242792    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:11.256451 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:11.256415    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:11.394491 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:11.394460    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:13:51.080704 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:51.080616    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"]
Sep 10 13:13:52.073931 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:52.073885    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"]
Sep 10 13:13:54.446315 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:13:54.446274    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:14:02.463045 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:02.463005    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:14:06.100760 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.100687    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp" podUID="78051822-5469-49e8-b13d-04c823cd9589" containerName="main" containerID="cri-o://e309929adc608ea6b63e28ca1da702771856e1b0beace79fea603082f6f64704" gracePeriod=45
Sep 10 13:14:06.363999 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.363922    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:14:06.489268 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.489224    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpdss\" (UniqueName: \"kubernetes.io/projected/78051822-5469-49e8-b13d-04c823cd9589-kube-api-access-lpdss\") pod \"78051822-5469-49e8-b13d-04c823cd9589\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") "
Sep 10 13:14:06.489488 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.489325    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-model-cache\") pod \"78051822-5469-49e8-b13d-04c823cd9589\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") "
Sep 10 13:14:06.489488 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.489377    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78051822-5469-49e8-b13d-04c823cd9589-tls-certs\") pod \"78051822-5469-49e8-b13d-04c823cd9589\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") "
Sep 10 13:14:06.489488 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.489419    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-home\") pod \"78051822-5469-49e8-b13d-04c823cd9589\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") "
Sep 10 13:14:06.489488 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.489446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-dshm\") pod \"78051822-5469-49e8-b13d-04c823cd9589\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") "
Sep 10 13:14:06.489763 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.489495    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-tmp-dir\") pod \"78051822-5469-49e8-b13d-04c823cd9589\" (UID: \"78051822-5469-49e8-b13d-04c823cd9589\") "
Sep 10 13:14:06.489763 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.489623    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-model-cache" (OuterVolumeSpecName: "model-cache") pod "78051822-5469-49e8-b13d-04c823cd9589" (UID: "78051822-5469-49e8-b13d-04c823cd9589"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:14:06.489763 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.489712    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-home" (OuterVolumeSpecName: "home") pod "78051822-5469-49e8-b13d-04c823cd9589" (UID: "78051822-5469-49e8-b13d-04c823cd9589"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:14:06.489897 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.489831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "78051822-5469-49e8-b13d-04c823cd9589" (UID: "78051822-5469-49e8-b13d-04c823cd9589"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:14:06.489897 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.489840    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:06.489897 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.489858    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:06.491711 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.491683    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-dshm" (OuterVolumeSpecName: "dshm") pod "78051822-5469-49e8-b13d-04c823cd9589" (UID: "78051822-5469-49e8-b13d-04c823cd9589"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:14:06.491826 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.491701    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78051822-5469-49e8-b13d-04c823cd9589-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "78051822-5469-49e8-b13d-04c823cd9589" (UID: "78051822-5469-49e8-b13d-04c823cd9589"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:14:06.491826 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.491769    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78051822-5469-49e8-b13d-04c823cd9589-kube-api-access-lpdss" (OuterVolumeSpecName: "kube-api-access-lpdss") pod "78051822-5469-49e8-b13d-04c823cd9589" (UID: "78051822-5469-49e8-b13d-04c823cd9589"). InnerVolumeSpecName "kube-api-access-lpdss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:14:06.591020 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.590961    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78051822-5469-49e8-b13d-04c823cd9589-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:06.591020 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.591010    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:06.591020 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.591021    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78051822-5469-49e8-b13d-04c823cd9589-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:06.591020 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.591030    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpdss\" (UniqueName: \"kubernetes.io/projected/78051822-5469-49e8-b13d-04c823cd9589-kube-api-access-lpdss\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:06.687371 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.687331    2563 generic.go:356] "Generic (PLEG): container finished" podID="78051822-5469-49e8-b13d-04c823cd9589" containerID="e309929adc608ea6b63e28ca1da702771856e1b0beace79fea603082f6f64704" exitCode=0
Sep 10 13:14:06.687618 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.687383    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"
Sep 10 13:14:06.687618 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.687414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp" event={"ID":"78051822-5469-49e8-b13d-04c823cd9589","Type":"ContainerDied","Data":"e309929adc608ea6b63e28ca1da702771856e1b0beace79fea603082f6f64704"}
Sep 10 13:14:06.687618 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.687453    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp" event={"ID":"78051822-5469-49e8-b13d-04c823cd9589","Type":"ContainerDied","Data":"badc1323aec9feefb6bd9b05d49bb7a06c33e7d4a543c5e2b69bc3023ee4dbf4"}
Sep 10 13:14:06.687618 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.687472    2563 scope.go:117] "RemoveContainer" containerID="e309929adc608ea6b63e28ca1da702771856e1b0beace79fea603082f6f64704"
Sep 10 13:14:06.697130 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.697112    2563 scope.go:117] "RemoveContainer" containerID="e309929adc608ea6b63e28ca1da702771856e1b0beace79fea603082f6f64704"
Sep 10 13:14:06.697463 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:14:06.697441    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e309929adc608ea6b63e28ca1da702771856e1b0beace79fea603082f6f64704\": container with ID starting with e309929adc608ea6b63e28ca1da702771856e1b0beace79fea603082f6f64704 not found: ID does not exist" containerID="e309929adc608ea6b63e28ca1da702771856e1b0beace79fea603082f6f64704"
Sep 10 13:14:06.697537 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.697471    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e309929adc608ea6b63e28ca1da702771856e1b0beace79fea603082f6f64704"} err="failed to get container status \"e309929adc608ea6b63e28ca1da702771856e1b0beace79fea603082f6f64704\": rpc error: code = NotFound desc = could not find container \"e309929adc608ea6b63e28ca1da702771856e1b0beace79fea603082f6f64704\": container with ID starting with e309929adc608ea6b63e28ca1da702771856e1b0beace79fea603082f6f64704 not found: ID does not exist"
Sep 10 13:14:06.733803 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.733762    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"]
Sep 10 13:14:06.754249 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:06.754213    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-43506975-kserve-54b79dd59-49rsp"]
Sep 10 13:14:07.094643 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.094492    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr" podUID="dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e" containerName="main" containerID="cri-o://57687afd2c90f84c05dc9230843310b059e12e62d2d6c978361b607574edfada" gracePeriod=45
Sep 10 13:14:07.362603 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.362567    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:14:07.498443 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.498398    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-home\") pod \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") "
Sep 10 13:14:07.498673 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.498524    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-tmp-dir\") pod \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") "
Sep 10 13:14:07.498673 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.498590    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-model-cache\") pod \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") "
Sep 10 13:14:07.498673 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.498620    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-tls-certs\") pod \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") "
Sep 10 13:14:07.498862 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.498705    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-dshm\") pod \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") "
Sep 10 13:14:07.498862 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.498721    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-home" (OuterVolumeSpecName: "home") pod "dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e" (UID: "dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:14:07.498862 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.498750    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9brlg\" (UniqueName: \"kubernetes.io/projected/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-kube-api-access-9brlg\") pod \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\" (UID: \"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e\") "
Sep 10 13:14:07.498862 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.498846    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e" (UID: "dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:14:07.499064 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.498858    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-model-cache" (OuterVolumeSpecName: "model-cache") pod "dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e" (UID: "dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:14:07.499108 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.499083    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-tmp-dir\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:07.499144 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.499109    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-model-cache\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:07.499144 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.499120    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-home\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:07.500954 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.500922    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-kube-api-access-9brlg" (OuterVolumeSpecName: "kube-api-access-9brlg") pod "dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e" (UID: "dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e"). InnerVolumeSpecName "kube-api-access-9brlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:14:07.501321 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.501293    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e" (UID: "dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:14:07.501321 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.501300    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-dshm" (OuterVolumeSpecName: "dshm") pod "dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e" (UID: "dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:14:07.600095 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.599991    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-dshm\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:07.600095 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.600033    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brlg\" (UniqueName: \"kubernetes.io/projected/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-kube-api-access-9brlg\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:07.600095 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.600045    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e-tls-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:07.697736 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.695453    2563 generic.go:356] "Generic (PLEG): container finished" podID="dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e" containerID="57687afd2c90f84c05dc9230843310b059e12e62d2d6c978361b607574edfada" exitCode=0
Sep 10 13:14:07.697736 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.695665    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"
Sep 10 13:14:07.698008 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.697734    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr" event={"ID":"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e","Type":"ContainerDied","Data":"57687afd2c90f84c05dc9230843310b059e12e62d2d6c978361b607574edfada"}
Sep 10 13:14:07.698008 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.697795    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr" event={"ID":"dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e","Type":"ContainerDied","Data":"c485ed4a079c41b686f8702d62d6e88ff52a03540e9079b6356f4c8e857bf3d4"}
Sep 10 13:14:07.698116 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.698021    2563 scope.go:117] "RemoveContainer" containerID="57687afd2c90f84c05dc9230843310b059e12e62d2d6c978361b607574edfada"
Sep 10 13:14:07.708191 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.708168    2563 scope.go:117] "RemoveContainer" containerID="57687afd2c90f84c05dc9230843310b059e12e62d2d6c978361b607574edfada"
Sep 10 13:14:07.708505 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:14:07.708487    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57687afd2c90f84c05dc9230843310b059e12e62d2d6c978361b607574edfada\": container with ID starting with 57687afd2c90f84c05dc9230843310b059e12e62d2d6c978361b607574edfada not found: ID does not exist" containerID="57687afd2c90f84c05dc9230843310b059e12e62d2d6c978361b607574edfada"
Sep 10 13:14:07.708611 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.708515    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57687afd2c90f84c05dc9230843310b059e12e62d2d6c978361b607574edfada"} err="failed to get container status \"57687afd2c90f84c05dc9230843310b059e12e62d2d6c978361b607574edfada\": rpc error: code = NotFound desc = could not find container \"57687afd2c90f84c05dc9230843310b059e12e62d2d6c978361b607574edfada\": container with ID starting with 57687afd2c90f84c05dc9230843310b059e12e62d2d6c978361b607574edfada not found: ID does not exist"
Sep 10 13:14:07.744579 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.744518    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"]
Sep 10 13:14:07.759209 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:07.759173    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-43506975-kserve-7b98f4dbf5-n7czr"]
Sep 10 13:14:08.704500 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:08.704462    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78051822-5469-49e8-b13d-04c823cd9589" path="/var/lib/kubelet/pods/78051822-5469-49e8-b13d-04c823cd9589/volumes"
Sep 10 13:14:08.704971 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:08.704954    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e" path="/var/lib/kubelet/pods/dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e/volumes"
Sep 10 13:14:09.897153 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.897110    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:14:09.952878 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.952830    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"]
Sep 10 13:14:09.953309 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.953292    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78051822-5469-49e8-b13d-04c823cd9589" containerName="main"
Sep 10 13:14:09.953357 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.953311    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="78051822-5469-49e8-b13d-04c823cd9589" containerName="main"
Sep 10 13:14:09.953357 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.953331    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e" containerName="main"
Sep 10 13:14:09.953357 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.953337    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e" containerName="main"
Sep 10 13:14:09.953456 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.953402    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="78051822-5469-49e8-b13d-04c823cd9589" containerName="main"
Sep 10 13:14:09.953456 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.953413    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6c860e-5ccd-4f65-8ce3-82cef1b54b9e" containerName="main"
Sep 10 13:14:09.958028 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.957994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:09.962611 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.962579    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 13:14:09.962611 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.962582    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 13:14:09.963426 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.963399    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 13:14:09.963541 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.963427    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 13:14:09.963541 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.963428    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xshvt\""
Sep 10 13:14:09.989862 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:09.989830    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"]
Sep 10 13:14:10.024033 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.023985    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-maas-api-tls\") pod \"maas-api-7797f985f6-zzxll\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") " pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:10.024033 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.024035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-metrics-certs\") pod \"maas-api-7797f985f6-zzxll\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") " pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:10.024257 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.024090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-openshift-service-ca\") pod \"maas-api-7797f985f6-zzxll\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") " pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:10.024257 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.024196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4kql\" (UniqueName: \"kubernetes.io/projected/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-kube-api-access-n4kql\") pod \"maas-api-7797f985f6-zzxll\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") " pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:10.125436 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.125387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-maas-api-tls\") pod \"maas-api-7797f985f6-zzxll\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") " pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:10.125436 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.125434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-metrics-certs\") pod \"maas-api-7797f985f6-zzxll\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") " pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:10.125699 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.125591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-openshift-service-ca\") pod \"maas-api-7797f985f6-zzxll\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") " pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:10.125739 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.125695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4kql\" (UniqueName: \"kubernetes.io/projected/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-kube-api-access-n4kql\") pod \"maas-api-7797f985f6-zzxll\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") " pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:10.126235 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.126210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-openshift-service-ca\") pod \"maas-api-7797f985f6-zzxll\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") " pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:10.128110 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.128084    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-maas-api-tls\") pod \"maas-api-7797f985f6-zzxll\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") " pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:10.128220 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.128127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-metrics-certs\") pod \"maas-api-7797f985f6-zzxll\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") " pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:10.149744 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.149660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4kql\" (UniqueName: \"kubernetes.io/projected/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-kube-api-access-n4kql\") pod \"maas-api-7797f985f6-zzxll\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") " pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:10.269424 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.269376    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:10.572580 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.572528    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"]
Sep 10 13:14:10.574481 ip-10-0-133-150 kubenswrapper[2563]: W0910 13:14:10.574455    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112ef564_6d4a_4c6e_aca1_f5c1b5fdca3d.slice/crio-f5453dd21b6a963572dfa18a6ba13a24c6de3bb27472338cf6c22e42010fc68f WatchSource:0}: Error finding container f5453dd21b6a963572dfa18a6ba13a24c6de3bb27472338cf6c22e42010fc68f: Status 404 returned error can't find the container with id f5453dd21b6a963572dfa18a6ba13a24c6de3bb27472338cf6c22e42010fc68f
Sep 10 13:14:10.575868 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.575844    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:14:10.720282 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:10.720242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll" event={"ID":"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d","Type":"ContainerStarted","Data":"f5453dd21b6a963572dfa18a6ba13a24c6de3bb27472338cf6c22e42010fc68f"}
Sep 10 13:14:12.733322 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:12.733279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll" event={"ID":"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d","Type":"ContainerStarted","Data":"4ecf8741137be6c3f5f1c111d0c5cc2c4d1791988fe854cdcfc59bb96df895cc"}
Sep 10 13:14:12.733793 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:12.733403    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:12.776585 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:12.776489    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll" podStartSLOduration=2.52751715 podStartE2EDuration="3.776472175s" totalImagesPullingTime="1.248955025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:14:12.775374936 +0000 UTC m=+2626.700286425" watchObservedRunningTime="2026-09-10 13:14:12.776472175 +0000 UTC m=+2626.701383662"
Sep 10 13:14:17.658229 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:17.658191    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:14:17.968933 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:17.968891    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"]
Sep 10 13:14:17.969154 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:17.969083    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll" podUID="112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d" containerName="maas-api" containerID="cri-o://4ecf8741137be6c3f5f1c111d0c5cc2c4d1791988fe854cdcfc59bb96df895cc" gracePeriod=30
Sep 10 13:14:17.974869 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:17.974841    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:18.109480 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.109437    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"]
Sep 10 13:14:18.114375 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.114341    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.143746 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.143709    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"]
Sep 10 13:14:18.207190 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.207154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34d433c9-fa7c-4346-9d7e-4b7c1d94c18a-metrics-certs\") pod \"maas-api-b4867cdb8-d9qgh\" (UID: \"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a\") " pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.207397 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.207217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d433c9-fa7c-4346-9d7e-4b7c1d94c18a-openshift-service-ca\") pod \"maas-api-b4867cdb8-d9qgh\" (UID: \"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a\") " pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.207397 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.207275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swf5j\" (UniqueName: \"kubernetes.io/projected/34d433c9-fa7c-4346-9d7e-4b7c1d94c18a-kube-api-access-swf5j\") pod \"maas-api-b4867cdb8-d9qgh\" (UID: \"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a\") " pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.207397 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.207372    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34d433c9-fa7c-4346-9d7e-4b7c1d94c18a-maas-api-tls\") pod \"maas-api-b4867cdb8-d9qgh\" (UID: \"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a\") " pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.243709 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.243675    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:18.308630 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.308591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d433c9-fa7c-4346-9d7e-4b7c1d94c18a-openshift-service-ca\") pod \"maas-api-b4867cdb8-d9qgh\" (UID: \"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a\") " pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.308838 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.308646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swf5j\" (UniqueName: \"kubernetes.io/projected/34d433c9-fa7c-4346-9d7e-4b7c1d94c18a-kube-api-access-swf5j\") pod \"maas-api-b4867cdb8-d9qgh\" (UID: \"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a\") " pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.308838 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.308712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34d433c9-fa7c-4346-9d7e-4b7c1d94c18a-maas-api-tls\") pod \"maas-api-b4867cdb8-d9qgh\" (UID: \"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a\") " pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.308838 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.308821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34d433c9-fa7c-4346-9d7e-4b7c1d94c18a-metrics-certs\") pod \"maas-api-b4867cdb8-d9qgh\" (UID: \"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a\") " pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.310093 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.309900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34d433c9-fa7c-4346-9d7e-4b7c1d94c18a-openshift-service-ca\") pod \"maas-api-b4867cdb8-d9qgh\" (UID: \"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a\") " pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.312249 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.312189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34d433c9-fa7c-4346-9d7e-4b7c1d94c18a-metrics-certs\") pod \"maas-api-b4867cdb8-d9qgh\" (UID: \"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a\") " pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.312386 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.312326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34d433c9-fa7c-4346-9d7e-4b7c1d94c18a-maas-api-tls\") pod \"maas-api-b4867cdb8-d9qgh\" (UID: \"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a\") " pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.322955 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.322918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swf5j\" (UniqueName: \"kubernetes.io/projected/34d433c9-fa7c-4346-9d7e-4b7c1d94c18a-kube-api-access-swf5j\") pod \"maas-api-b4867cdb8-d9qgh\" (UID: \"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a\") " pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.409441 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.409400    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-maas-api-tls\") pod \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") "
Sep 10 13:14:18.409633 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.409510    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4kql\" (UniqueName: \"kubernetes.io/projected/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-kube-api-access-n4kql\") pod \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") "
Sep 10 13:14:18.409633 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.409574    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-metrics-certs\") pod \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") "
Sep 10 13:14:18.409633 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.409609    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-openshift-service-ca\") pod \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\" (UID: \"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d\") "
Sep 10 13:14:18.410127 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.410093    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d" (UID: "112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:14:18.411864 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.411830    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-kube-api-access-n4kql" (OuterVolumeSpecName: "kube-api-access-n4kql") pod "112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d" (UID: "112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d"). InnerVolumeSpecName "kube-api-access-n4kql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:14:18.412121 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.412092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d" (UID: "112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:14:18.412121 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.412105    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d" (UID: "112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:14:18.428515 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.428474    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:18.511304 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.511262    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-maas-api-tls\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:18.511304 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.511301    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4kql\" (UniqueName: \"kubernetes.io/projected/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-kube-api-access-n4kql\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:18.511304 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.511311    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-metrics-certs\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:18.511564 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.511322    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d-openshift-service-ca\") on node \"ip-10-0-133-150.ec2.internal\" DevicePath \"\""
Sep 10 13:14:18.735334 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.735247    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"]
Sep 10 13:14:18.738484 ip-10-0-133-150 kubenswrapper[2563]: W0910 13:14:18.738453    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d433c9_fa7c_4346_9d7e_4b7c1d94c18a.slice/crio-f93b3d53c76cedce5f11d403f7c8eec68e80633ae00d791994d509cf9a16ceed WatchSource:0}: Error finding container f93b3d53c76cedce5f11d403f7c8eec68e80633ae00d791994d509cf9a16ceed: Status 404 returned error can't find the container with id f93b3d53c76cedce5f11d403f7c8eec68e80633ae00d791994d509cf9a16ceed
Sep 10 13:14:18.773729 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.773686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh" event={"ID":"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a","Type":"ContainerStarted","Data":"f93b3d53c76cedce5f11d403f7c8eec68e80633ae00d791994d509cf9a16ceed"}
Sep 10 13:14:18.775486 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.775448    2563 generic.go:356] "Generic (PLEG): container finished" podID="112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d" containerID="4ecf8741137be6c3f5f1c111d0c5cc2c4d1791988fe854cdcfc59bb96df895cc" exitCode=0
Sep 10 13:14:18.775699 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.775500    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"
Sep 10 13:14:18.775699 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.775511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll" event={"ID":"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d","Type":"ContainerDied","Data":"4ecf8741137be6c3f5f1c111d0c5cc2c4d1791988fe854cdcfc59bb96df895cc"}
Sep 10 13:14:18.775699 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.775585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7797f985f6-zzxll" event={"ID":"112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d","Type":"ContainerDied","Data":"f5453dd21b6a963572dfa18a6ba13a24c6de3bb27472338cf6c22e42010fc68f"}
Sep 10 13:14:18.775699 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.775617    2563 scope.go:117] "RemoveContainer" containerID="4ecf8741137be6c3f5f1c111d0c5cc2c4d1791988fe854cdcfc59bb96df895cc"
Sep 10 13:14:18.794150 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.794108    2563 scope.go:117] "RemoveContainer" containerID="4ecf8741137be6c3f5f1c111d0c5cc2c4d1791988fe854cdcfc59bb96df895cc"
Sep 10 13:14:18.794642 ip-10-0-133-150 kubenswrapper[2563]: E0910 13:14:18.794612    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ecf8741137be6c3f5f1c111d0c5cc2c4d1791988fe854cdcfc59bb96df895cc\": container with ID starting with 4ecf8741137be6c3f5f1c111d0c5cc2c4d1791988fe854cdcfc59bb96df895cc not found: ID does not exist" containerID="4ecf8741137be6c3f5f1c111d0c5cc2c4d1791988fe854cdcfc59bb96df895cc"
Sep 10 13:14:18.794765 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.794657    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecf8741137be6c3f5f1c111d0c5cc2c4d1791988fe854cdcfc59bb96df895cc"} err="failed to get container status \"4ecf8741137be6c3f5f1c111d0c5cc2c4d1791988fe854cdcfc59bb96df895cc\": rpc error: code = NotFound desc = could not find container \"4ecf8741137be6c3f5f1c111d0c5cc2c4d1791988fe854cdcfc59bb96df895cc\": container with ID starting with 4ecf8741137be6c3f5f1c111d0c5cc2c4d1791988fe854cdcfc59bb96df895cc not found: ID does not exist"
Sep 10 13:14:18.812589 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.812525    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"]
Sep 10 13:14:18.828412 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:18.828374    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7797f985f6-zzxll"]
Sep 10 13:14:20.705782 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:20.705746    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d" path="/var/lib/kubelet/pods/112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d/volumes"
Sep 10 13:14:20.789686 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:20.789643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh" event={"ID":"34d433c9-fa7c-4346-9d7e-4b7c1d94c18a","Type":"ContainerStarted","Data":"20b3ac465c63fa39a8679162edb2e4b66baba5dd136de0a265687f280068e909"}
Sep 10 13:14:20.789842 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:20.789713    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:20.823098 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:20.823037    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh" podStartSLOduration=1.276114317 podStartE2EDuration="2.823018531s" totalImagesPullingTime="1.546904214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:14:20.821612606 +0000 UTC m=+2634.746524094" watchObservedRunningTime="2026-09-10 13:14:20.823018531 +0000 UTC m=+2634.747930018"
Sep 10 13:14:25.150042 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:25.150005    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:14:26.801456 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:26.801422    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b4867cdb8-d9qgh"
Sep 10 13:14:31.281076 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:31.281029    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:14:37.248374 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:37.248331    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-tm276"]
Sep 10 13:14:47.541095 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:47.541061    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jx7nw_87452133-eecf-4359-b5bd-a7b37c5750c8/manager/0.log"
Sep 10 13:14:47.676298 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:47.676254    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rpnjj_2c7804bb-3e5c-4105-9ab1-262b3a349561/manager/0.log"
Sep 10 13:14:47.973250 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:47.973222    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jxq4n_05f31c4d-8948-4332-bc14-40e5a0968652/server/0.log"
Sep 10 13:14:48.222675 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:48.222640    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-669b9dff44-5cfwp_48f3d0a0-8c15-4ffe-a104-34fa94267f47/copy-manifests/0.log"
Sep 10 13:14:48.241311 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:48.241240    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-669b9dff44-5cfwp_48f3d0a0-8c15-4ffe-a104-34fa94267f47/manager/0.log"
Sep 10 13:14:48.949274 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:48.949245    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-b4867cdb8-d9qgh_34d433c9-fa7c-4346-9d7e-4b7c1d94c18a/maas-api/0.log"
Sep 10 13:14:49.209037 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:49.208957    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817405-bcqjp_2da7d2c3-3e1f-4736-ac40-7f2a49ed8f8c/cleanup/0.log"
Sep 10 13:14:49.322924 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:49.322893    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817420-ft4wb_507dec08-5692-47e1-9c83-e632aed73127/cleanup/0.log"
Sep 10 13:14:50.470724 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:50.470694    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-bgjwx_aa9b2d36-210f-4c0f-82c4-cb5a233c29ef/manager/0.log"
Sep 10 13:14:50.594625 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:50.594590    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-clc6x_5b880597-db64-45fa-bcdb-2ed1808548d1/manager/0.log"
Sep 10 13:14:50.706639 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:50.706611    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-nww57_b6765bc4-ff39-4903-b62b-caae6d75bd72/kuadrant-console-plugin/0.log"
Sep 10 13:14:50.845392 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:50.845302    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-hd9d5_e9cefaa6-82f9-4852-86f8-7eda40fb06d6/manager/0.log"
Sep 10 13:14:50.982081 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:50.982053    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tm276_b61d0759-f7e6-4600-9f93-2b55f09edf54/limitador/0.log"
Sep 10 13:14:51.111515 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:51.111422    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-92vhl_cdc478b1-236b-4e43-9914-9826ce71ef44/manager/0.log"
Sep 10 13:14:51.726642 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:51.726607    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-669df89dc4-gzv5p_6fd4b376-afab-4151-a2a8-1dd23c901612/kube-auth-proxy/0.log"
Sep 10 13:14:51.964754 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:51.964723    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tsqwk_dca38e13-c602-4bf5-8f9f-845c37b7ab9d/istio-proxy/1.log"
Sep 10 13:14:52.987767 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:52.987738    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-pmp55_44e48684-dba7-494f-9bc0-378b86c8bb4b/main/0.log"
Sep 10 13:14:53.348589 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:53.348480    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-5gtcm_c4a9a835-70a1-48f1-bbed-8c7847263caa/main/0.log"
Sep 10 13:14:53.464189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:14:53.464157    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-tg2ks_53652eb5-0e3a-47a3-ae74-eeff17a0dcdc/main/0.log"
Sep 10 13:15:03.078585 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:03.078532    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jfpm9_4be22799-8b7d-4e18-867f-34a3185af15a/global-pull-secret-syncer/0.log"
Sep 10 13:15:03.342403 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:03.342312    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sc5dh_851a86dd-ec09-426e-a81e-71c143dacb4f/konnectivity-agent/0.log"
Sep 10 13:15:03.456939 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:03.456908    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-150.ec2.internal_87b3b84e20b1589782e644bb9f5ff7c5/haproxy/0.log"
Sep 10 13:15:08.268075 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:08.268040    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-bgjwx_aa9b2d36-210f-4c0f-82c4-cb5a233c29ef/manager/0.log"
Sep 10 13:15:08.304414 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:08.304382    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-clc6x_5b880597-db64-45fa-bcdb-2ed1808548d1/manager/0.log"
Sep 10 13:15:08.348433 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:08.348397    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-nww57_b6765bc4-ff39-4903-b62b-caae6d75bd72/kuadrant-console-plugin/0.log"
Sep 10 13:15:08.420090 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:08.420055    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-hd9d5_e9cefaa6-82f9-4852-86f8-7eda40fb06d6/manager/0.log"
Sep 10 13:15:08.509750 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:08.509721    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-tm276_b61d0759-f7e6-4600-9f93-2b55f09edf54/limitador/0.log"
Sep 10 13:15:08.590699 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:08.590612    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-92vhl_cdc478b1-236b-4e43-9914-9826ce71ef44/manager/0.log"
Sep 10 13:15:10.230114 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:10.230084    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d773150f-ac5f-4997-8240-ac31df2cb1b6/alertmanager/0.log"
Sep 10 13:15:10.278029 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:10.277995    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d773150f-ac5f-4997-8240-ac31df2cb1b6/config-reloader/0.log"
Sep 10 13:15:10.311478 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:10.311449    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d773150f-ac5f-4997-8240-ac31df2cb1b6/kube-rbac-proxy-web/0.log"
Sep 10 13:15:10.340926 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:10.340895    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d773150f-ac5f-4997-8240-ac31df2cb1b6/kube-rbac-proxy/0.log"
Sep 10 13:15:10.377262 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:10.377178    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d773150f-ac5f-4997-8240-ac31df2cb1b6/kube-rbac-proxy-metric/0.log"
Sep 10 13:15:10.411706 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:10.411679    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d773150f-ac5f-4997-8240-ac31df2cb1b6/prom-label-proxy/0.log"
Sep 10 13:15:10.438463 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:10.438427    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d773150f-ac5f-4997-8240-ac31df2cb1b6/init-config-reloader/0.log"
Sep 10 13:15:10.491279 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:10.491244    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-nbc4r_b897d389-1590-4c48-9504-24c834416b86/cluster-monitoring-operator/0.log"
Sep 10 13:15:10.713154 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:10.713123    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-cg64p_30ea156b-7223-4fd1-9b04-2ffde135f99c/monitoring-plugin/0.log"
Sep 10 13:15:10.770938 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:10.770905    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77whj_1a139624-4246-47bd-acb1-d8acd9c5c272/node-exporter/0.log"
Sep 10 13:15:10.833319 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:10.833289    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77whj_1a139624-4246-47bd-acb1-d8acd9c5c272/kube-rbac-proxy/0.log"
Sep 10 13:15:10.886758 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:10.886728    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-77whj_1a139624-4246-47bd-acb1-d8acd9c5c272/init-textfile/0.log"
Sep 10 13:15:12.138308 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:12.138217    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-svtzz_a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9/prometheus-operator/0.log"
Sep 10 13:15:12.206194 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:12.206167    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-svtzz_a0f9d19a-ba01-4f53-9e2f-ba110c9b6fc9/kube-rbac-proxy/0.log"
Sep 10 13:15:12.510444 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:12.510408    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cd78b988c-snz6p_41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc/thanos-query/0.log"
Sep 10 13:15:12.552049 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:12.552016    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cd78b988c-snz6p_41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc/kube-rbac-proxy-web/0.log"
Sep 10 13:15:12.591699 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:12.591669    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cd78b988c-snz6p_41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc/kube-rbac-proxy/0.log"
Sep 10 13:15:12.640901 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:12.640861    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cd78b988c-snz6p_41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc/prom-label-proxy/0.log"
Sep 10 13:15:12.690256 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:12.690225    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cd78b988c-snz6p_41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc/kube-rbac-proxy-rules/0.log"
Sep 10 13:15:12.718230 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:12.718198    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7cd78b988c-snz6p_41c0d533-a07e-49bc-a7a5-8ea2e97e3dcc/kube-rbac-proxy-metrics/0.log"
Sep 10 13:15:13.476862 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.476828    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"]
Sep 10 13:15:13.477291 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.477283    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d" containerName="maas-api"
Sep 10 13:15:13.477335 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.477296    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d" containerName="maas-api"
Sep 10 13:15:13.477371 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.477364    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="112ef564-6d4a-4c6e-aca1-f5c1b5fdca3d" containerName="maas-api"
Sep 10 13:15:13.480613 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.480585    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.483530 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.483503    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fkh8h\"/\"kube-root-ca.crt\""
Sep 10 13:15:13.484450 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.484414    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fkh8h\"/\"default-dockercfg-dskqh\""
Sep 10 13:15:13.484450 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.484439    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fkh8h\"/\"openshift-service-ca.crt\""
Sep 10 13:15:13.495270 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.495230    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"]
Sep 10 13:15:13.554666 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.554624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzngb\" (UniqueName: \"kubernetes.io/projected/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-kube-api-access-jzngb\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.554666 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.554668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-sys\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.554902 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.554698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-podres\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.554902 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.554775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-lib-modules\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.554902 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.554829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-proc\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.655542 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.655504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-lib-modules\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.655781 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.655594    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-proc\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.655781 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.655662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzngb\" (UniqueName: \"kubernetes.io/projected/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-kube-api-access-jzngb\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.655781 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.655685    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-sys\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.655781 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.655707    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-podres\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.655781 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.655721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-proc\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.655781 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.655722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-lib-modules\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.656081 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.655785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-sys\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.656081 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.655827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-podres\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.668228 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.668183    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzngb\" (UniqueName: \"kubernetes.io/projected/80a3635b-c9f7-4fd2-b5cd-c385bcc967a2-kube-api-access-jzngb\") pod \"perf-node-gather-daemonset-dh46t\" (UID: \"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2\") " pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:13.791713 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:13.791616    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:14.004391 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:14.004340    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-ktd7n_a8cf6d4b-edf0-48a6-8bfa-26aab374b6f9/networking-console-plugin/0.log"
Sep 10 13:15:14.088331 ip-10-0-133-150 kubenswrapper[2563]: W0910 13:15:14.088293    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod80a3635b_c9f7_4fd2_b5cd_c385bcc967a2.slice/crio-71011827e3dcd81a54f063b230f00fdcede972e7fe83f6ca00bba69c9cf1d7ee WatchSource:0}: Error finding container 71011827e3dcd81a54f063b230f00fdcede972e7fe83f6ca00bba69c9cf1d7ee: Status 404 returned error can't find the container with id 71011827e3dcd81a54f063b230f00fdcede972e7fe83f6ca00bba69c9cf1d7ee
Sep 10 13:15:14.088748 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:14.088683    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"]
Sep 10 13:15:15.098890 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:15.098846    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t" event={"ID":"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2","Type":"ContainerStarted","Data":"255a0e086cd48fd8e787961f887868ea6e4658d8e13dc9ddc30df45f6a28c918"}
Sep 10 13:15:15.099345 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:15.098902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t" event={"ID":"80a3635b-c9f7-4fd2-b5cd-c385bcc967a2","Type":"ContainerStarted","Data":"71011827e3dcd81a54f063b230f00fdcede972e7fe83f6ca00bba69c9cf1d7ee"}
Sep 10 13:15:15.099345 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:15.099054    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:15.129342 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:15.129280    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t" podStartSLOduration=2.129261651 podStartE2EDuration="2.129261651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:15:15.128463673 +0000 UTC m=+2689.053375162" watchObservedRunningTime="2026-09-10 13:15:15.129261651 +0000 UTC m=+2689.054173139"
Sep 10 13:15:17.198929 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:17.198900    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5qwhw_1bf1ff93-c28e-404e-b894-b82f73484d09/dns/0.log"
Sep 10 13:15:17.296199 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:17.296168    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5qwhw_1bf1ff93-c28e-404e-b894-b82f73484d09/kube-rbac-proxy/0.log"
Sep 10 13:15:17.632759 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:17.632677    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j7fz9_a6e7308e-da70-4a2b-845a-51057e2f1f94/dns-node-resolver/0.log"
Sep 10 13:15:18.302833 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:18.302790    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64fb4d4586-kbrd6_614191f8-6294-44de-b18f-37d7f71e8e0d/registry/0.log"
Sep 10 13:15:18.364367 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:18.364338    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n5899_c1e745f6-42ce-443e-8555-1572dbad7050/node-ca/0.log"
Sep 10 13:15:19.907393 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:19.907363    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-669df89dc4-gzv5p_6fd4b376-afab-4151-a2a8-1dd23c901612/kube-auth-proxy/0.log"
Sep 10 13:15:20.025866 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:20.025827    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tsqwk_dca38e13-c602-4bf5-8f9f-845c37b7ab9d/istio-proxy/0.log"
Sep 10 13:15:20.028189 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:20.028167    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tsqwk_dca38e13-c602-4bf5-8f9f-845c37b7ab9d/istio-proxy/1.log"
Sep 10 13:15:21.124285 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:21.124254    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fkh8h/perf-node-gather-daemonset-dh46t"
Sep 10 13:15:21.589090 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:21.589057    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mfltw_e2576131-0632-4603-a98c-e47cd0c73e69/serve-healthcheck-canary/0.log"
Sep 10 13:15:23.395708 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:23.395678    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tqkdr_7a8ce703-8992-44dc-a56c-1313710a627d/kube-rbac-proxy/0.log"
Sep 10 13:15:23.446041 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:23.446009    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tqkdr_7a8ce703-8992-44dc-a56c-1313710a627d/exporter/0.log"
Sep 10 13:15:23.548252 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:23.548217    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tqkdr_7a8ce703-8992-44dc-a56c-1313710a627d/extractor/0.log"
Sep 10 13:15:23.740583 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:23.740529    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-669df89dc4-gzv5p_6fd4b376-afab-4151-a2a8-1dd23c901612/kube-auth-proxy/0.log"
Sep 10 13:15:23.820390 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:23.820295    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tsqwk_dca38e13-c602-4bf5-8f9f-845c37b7ab9d/istio-proxy/0.log"
Sep 10 13:15:23.822699 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:23.822676    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tsqwk_dca38e13-c602-4bf5-8f9f-845c37b7ab9d/istio-proxy/1.log"
Sep 10 13:15:26.081954 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:26.081921    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-669df89dc4-gzv5p_6fd4b376-afab-4151-a2a8-1dd23c901612/kube-auth-proxy/0.log"
Sep 10 13:15:26.135642 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:26.135482    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tsqwk_dca38e13-c602-4bf5-8f9f-845c37b7ab9d/istio-proxy/0.log"
Sep 10 13:15:26.137777 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:26.137756    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tsqwk_dca38e13-c602-4bf5-8f9f-845c37b7ab9d/istio-proxy/1.log"
Sep 10 13:15:26.717813 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:26.717783    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-jx7nw_87452133-eecf-4359-b5bd-a7b37c5750c8/manager/0.log"
Sep 10 13:15:27.009410 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:27.009313    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-rpnjj_2c7804bb-3e5c-4105-9ab1-262b3a349561/manager/0.log"
Sep 10 13:15:27.100050 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:27.100022    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-jxq4n_05f31c4d-8948-4332-bc14-40e5a0968652/server/0.log"
Sep 10 13:15:27.240916 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:27.240875    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-669b9dff44-5cfwp_48f3d0a0-8c15-4ffe-a104-34fa94267f47/manager/0.log"
Sep 10 13:15:27.272894 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:27.272795    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-669b9dff44-5cfwp_48f3d0a0-8c15-4ffe-a104-34fa94267f47/copy-manifests/0.log"
Sep 10 13:15:28.914013 ip-10-0-133-150 kubenswrapper[2563]: I0910 13:15:28.913981    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-nhzlv_1de3f6dc-4b41-43a5-b0fa-127f68713c57/openshift-lws-operator/0.log"