Sep 18 15:32:11.365919 ip-10-0-133-137 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 15:32:11.365933 ip-10-0-133-137 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 15:32:11.365943 ip-10-0-133-137 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 15:32:11.366256 ip-10-0-133-137 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 15:32:21.671904 ip-10-0-133-137 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 18 15:32:21.671917 ip-10-0-133-137 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 1186195cd9c8408b93545469a96abbd5 --
Sep 18 15:34:15.105927 ip-10-0-133-137 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 15:34:18.179808 ip-10-0-133-137 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 15:34:18.179808 ip-10-0-133-137 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 15:34:18.179808 ip-10-0-133-137 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 15:34:18.179808 ip-10-0-133-137 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 15:34:18.179808 ip-10-0-133-137 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 15:34:18.180462 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.180377    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 15:34:18.185464 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185449    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 15:34:18.185464 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185464    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185468    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185471    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185474    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185477    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185479    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185482    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185485    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185488    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185492    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185494    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185501    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185504    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185507    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185510    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185512    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185515    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185518    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185521    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185523    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 15:34:18.185530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185526    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185529    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185531    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185536    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185540    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185544    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185548    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185551    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185554    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185557    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185559    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185562    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185565    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185568    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185571    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185574    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185576    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185578    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185582    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 15:34:18.186015 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185585    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185588    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185591    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185594    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185597    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185600    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185602    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185605    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185608    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185610    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185613    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185615    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185618    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185620    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185623    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185626    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185629    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185633    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185635    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185638    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 15:34:18.186535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185640    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185643    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185645    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185648    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185650    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185653    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185655    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185672    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185676    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185680    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185684    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185688    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185692    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185694    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185697    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185699    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185702    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185707    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185710    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 15:34:18.187044 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185714    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185716    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185719    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185722    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185724    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185727    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185730    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185732    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185739    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185742    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185744    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185747    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185750    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185752    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185755    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185758    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185760    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185763    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185765    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.185768    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 15:34:18.187506 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186441    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186447    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186450    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186453    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186456    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186458    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186461    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186464    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186466    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186469    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186473    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186478    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186483    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186487    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186490    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186493    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186496    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186499    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186501    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 15:34:18.187995 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186504    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186509    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186511    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186514    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186517    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186520    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186523    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186525    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186528    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186530    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186533    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186536    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186539    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186542    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186545    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186548    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186551    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186553    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186556    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186559    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 15:34:18.188473 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186561    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186564    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186567    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186570    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186572    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186575    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186578    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186581    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186584    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186586    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186589    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186591    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186594    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186598    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186600    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186603    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186606    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186609    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186611    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186614    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 15:34:18.189004 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186616    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186619    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186622    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186624    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186627    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186630    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186632    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186635    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186638    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186640    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186643    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186645    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186648    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186651    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186653    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186657    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186674    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186677    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186682    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 15:34:18.189490 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186686    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186689    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186692    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186694    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186697    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186699    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186703    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186706    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186709    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186712    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186714    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186717    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186719    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186722    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186725    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186727    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186730    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186733    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186735    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186738    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 15:34:18.189972 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.186741    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187094    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187098    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187101    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187104    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187107    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187110    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187112    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187115    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187118    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187121    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187125    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187128    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187130    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187133    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187136    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187138    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187141    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187145    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 15:34:18.190467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187149    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187153    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187156    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187159    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187161    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187164    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187167    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187169    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187173    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187176    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187180    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187183    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187186    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187189    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187192    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187194    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187197    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187200    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187202    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 15:34:18.190945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187205    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187207    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187210    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187213    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187217    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187219    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187223    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187226    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187229    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187231    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187234    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187237    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187241    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187243    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187246    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187248    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187251    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187253    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187256    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 15:34:18.191406 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187258    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187261    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187263    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187266    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187269    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187271    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187274    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187276    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187279    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187281    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187284    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187286    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187289    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187292    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187294    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187296    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187299    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187302    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187304    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187307    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 15:34:18.191937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187310    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187313    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187316    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187319    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187322    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187326    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187328    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187331    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187333    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187336    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187339    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187342    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187344    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187347    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187349    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187352    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187354    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187357    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187360    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187362    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 15:34:18.192424 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187365    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187367    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.187370    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187442    2561 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187494    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187501    2561 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187505    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187510    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187513    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187517    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187522    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187525    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187529    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187533    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187536    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187540    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187543    2561 flags.go:64] FLAG: --cgroup-root=""
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187547    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187550    2561 flags.go:64] FLAG: --client-ca-file=""
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187553    2561 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187556    2561 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187561    2561 flags.go:64] FLAG: --cluster-domain=""
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187564    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187567    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 15:34:18.192915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187570    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187573    2561 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187577    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187580    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187583    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187587    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187590    2561 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187593    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187596    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187599    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187602    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187606    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187608    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187611    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187615    2561 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187618    2561 flags.go:64] FLAG: --enable-server="true"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187621    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187625    2561 flags.go:64] FLAG: --event-burst="100"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187628    2561 flags.go:64] FLAG: --event-qps="50"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187631    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187634    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187637    2561 flags.go:64] FLAG: --eviction-hard=""
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187641    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187644    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187647    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 15:34:18.193493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187653    2561 flags.go:64] FLAG: --eviction-soft=""
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187656    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187670    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187673    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187676    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187679    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187683    2561 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187685    2561 flags.go:64] FLAG: --feature-gates=""
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187689    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187692    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187695    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187698    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187701    2561 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187704    2561 flags.go:64] FLAG: --help="false"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187707    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187710    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187714    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187717    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187720    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187723    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187726    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187729    2561 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187732    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 15:34:18.194118 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187735    2561 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187737    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187741    2561 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187743    2561 flags.go:64] FLAG: --kube-reserved=""
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187746    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187749    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187752    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187755    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187758    2561 flags.go:64] FLAG: --lock-file=""
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187763    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187766    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187769    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187774    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187777    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187779    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187782    2561 flags.go:64] FLAG: --logging-format="text"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187785    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187788    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187791    2561 flags.go:64] FLAG: --manifest-url=""
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187794    2561 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187798    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187801    2561 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187805    2561 flags.go:64] FLAG: --max-pods="110"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187808    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187811    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 15:34:18.194838 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187815    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187818    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187821    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187824    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187827    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187834    2561 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187837    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187840    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187843    2561 flags.go:64] FLAG: --pod-cidr=""
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187846    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187852    2561 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187855    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187858    2561 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187860    2561 flags.go:64] FLAG: --port="10250"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187863    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187867    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02138514c6853a294"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187871    2561 flags.go:64] FLAG: --qos-reserved=""
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187875    2561 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187878    2561 flags.go:64] FLAG: --register-node="true"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187881    2561 flags.go:64] FLAG: --register-with-taints=""
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187884    2561 flags.go:64] FLAG: --registry-burst="10"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187887    2561 flags.go:64] FLAG: --registry-qps="5"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187890    2561 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187893    2561 flags.go:64] FLAG: --reserved-memory=""
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187897    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 15:34:18.195459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187900    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187903    2561 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187905    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187908    2561 flags.go:64] FLAG: --runonce="false"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187911    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187914    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187917    2561 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187921    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187924    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187927    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187930    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187933    2561 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187936    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187939    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187942    2561 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187945    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187948    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187950    2561 flags.go:64] FLAG: --system-cgroups=""
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187953    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187958    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187961    2561 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187964    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187968    2561 flags.go:64] FLAG: --tls-min-version=""
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187972    2561 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187977    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 15:34:18.196081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187980    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187983    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187986    2561 flags.go:64] FLAG: --v="2"
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187989    2561 flags.go:64] FLAG: --version="false"
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187993    2561 flags.go:64] FLAG: --vmodule=""
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.187997    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.188000    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188113    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188116    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188119    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188122    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188125    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188128    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188131    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188135    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188138    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188140    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188143    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188146    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188148    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188151    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 15:34:18.196784 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188154    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188156    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188159    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188161    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188164    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188166    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188169    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188172    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188174    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188177    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188181    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188184    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188186    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188189    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188192    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188195    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188198    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188200    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188203    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188205    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 15:34:18.197674 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188208    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188210    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188213    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188215    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188218    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188222    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188224    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188227    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188229    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188232    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188234    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188237    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188239    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188242    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188245    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188247    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188250    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188253    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188255    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188258    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 15:34:18.198203 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188260    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188263    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188266    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188269    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188272    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188274    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188277    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188279    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188282    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188285    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188287    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188290    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188292    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188295    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188298    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188302    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188305    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188309    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188312    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 15:34:18.198761 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188315    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188318    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188320    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188323    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188326    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188329    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188332    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188336    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188339    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188341    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188344    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188346    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188349    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188352    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188354    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188357    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188359    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188362    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188365    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 15:34:18.199215 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188368    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188370    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188373    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188375    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188380    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188383    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.188386    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.189121    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.197447    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.197525    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.197576    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.197586    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197858    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197875    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 15:34:18.199702 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197881    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197887    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197892    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197897    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197901    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197906    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197910    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197914    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197918    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197922    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197926    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197930    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197934    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197938    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197942    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197946    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197951    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197955    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197959    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197964    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 15:34:18.200097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197968    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197972    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197976    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197980    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197985    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197994    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.197998    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198003    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198007    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198012    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198016    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198021    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198024    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198027    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198030    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198033    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198035    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198038    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198041    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198043    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 15:34:18.200582 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198046    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198049    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198056    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198062    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198066    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198070    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198080    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198084    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198087    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198090    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198093    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198096    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198098    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198101    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198104    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198107    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198109    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198112    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198115    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 15:34:18.201079 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198118    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198121    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198123    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198127    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198130    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198133    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198136    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198139    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198142    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198144    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198147    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198150    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198153    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198155    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198158    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198161    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198163    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198166    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198168    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198171    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 15:34:18.201542 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198173    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198176    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198179    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198181    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198184    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198186    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198189    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198192    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198194    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198197    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198199    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198202    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198205    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198208    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198210    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198214    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198217    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 15:34:18.202124 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198220    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.198225    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198344    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198349    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198352    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198355    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198358    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198361    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198364    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198367    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198369    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198372    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198375    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198377    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198380    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 15:34:18.202548 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198383    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198386    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198388    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198391    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198394    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198396    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198399    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198402    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198404    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198407    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198410    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198412    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198415    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198418    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198420    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198423    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198426    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198429    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198431    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198434    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 15:34:18.202937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198436    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198439    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198442    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198444    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198447    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198450    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198452    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198455    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198457    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198460    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198462    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198465    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198474    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198477    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198479    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198482    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198484    2561 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198487    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198490    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 15:34:18.203418 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198493    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198498    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198502    2561 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198506    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198509    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198512    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198516    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198519    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198522    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198525    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198527    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198530    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198532    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198535    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198537    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198540    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198542    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198545    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198547    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 15:34:18.203957 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198550    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198552    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198555    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198557    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198561    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198565    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198567    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198571    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198574    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198577    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198580    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198583    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198586    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198588    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198591    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198594    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198597    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198599    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198602    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198605    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 15:34:18.204439 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198607    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 15:34:18.204937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198610    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 15:34:18.204937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198613    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 15:34:18.204937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198615    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 15:34:18.204937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198618    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 15:34:18.204937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198620    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 15:34:18.204937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198623    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 15:34:18.204937 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:18.198625    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 15:34:18.204937 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.198630    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 15:34:18.204937 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.198750    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 15:34:18.204937 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.201069    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 15:34:18.204937 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.201723    2561 server.go:1019] "Starting client certificate rotation"
Sep 18 15:34:18.204937 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.201818    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 15:34:18.204937 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.201847    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 15:34:18.205253 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.204098    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 15:34:18.205514 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.205499    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 15:34:18.211069 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.211052    2561 log.go:25] "Validated CRI v1 runtime API"
Sep 18 15:34:18.212508 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.212497    2561 log.go:25] "Validated CRI v1 image API"
Sep 18 15:34:18.213041 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.213030    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 15:34:18.215733 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.215713    2561 fs.go:135] Filesystem UUIDs: map[3a12c2f4-72dd-44e4-85e5-aab505d79f64:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 943bfe8e-8c84-439e-9e4f-4b9d362a308c:/dev/nvme0n1p4]
Sep 18 15:34:18.215822 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.215730    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 15:34:18.220154 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.220051    2561 manager.go:216] Machine: {Timestamp:2026-09-18 15:34:18.219549034 +0000 UTC m=+1.764839700 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097608 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21ddc20a7346962847b9865e0f5152 SystemUUID:ec21ddc2-0a73-4696-2847-b9865e0f5152 BootID:1186195c-d9c8-408b-9354-5469a96abbd5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:11:3c:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:11:3c:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:40:f9:2b:ee:89 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 15:34:18.220154 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.220147    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 15:34:18.220275 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.220217    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 15:34:18.221671 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.221633    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 15:34:18.221824 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.221679    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-137.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 15:34:18.222849 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.222835    2561 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 15:34:18.222883 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.222853    2561 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 15:34:18.222883 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.222866    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 15:34:18.222883 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.222879    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 15:34:18.222963 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.222944    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 15:34:18.225944 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.225931    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 15:34:18.226069 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.226059    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 15:34:18.226124 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.226105    2561 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 15:34:18.226124 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.226115    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 15:34:18.226190 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.226125    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 15:34:18.226190 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.226133    2561 kubelet.go:398] "Adding apiserver pod source"
Sep 18 15:34:18.226190 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.226142    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 15:34:18.226539 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.226522    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 15:34:18.226594 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.226549    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 15:34:18.229477 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.229410    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 15:34:18.239599 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.239582    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 15:34:18.239695 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.239613    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 15:34:18.240776 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240761    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 15:34:18.240832 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240787    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 15:34:18.240832 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240796    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 15:34:18.240832 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240804    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 15:34:18.240832 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240815    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 15:34:18.240832 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240823    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 15:34:18.241011 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240841    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 15:34:18.241011 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240849    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 15:34:18.241011 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240859    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 15:34:18.241011 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240867    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 15:34:18.241011 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240879    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 15:34:18.241011 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240886    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 15:34:18.241011 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240916    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 15:34:18.241011 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.240923    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 15:34:18.243683 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.243647    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 15:34:18.243761 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.243671    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-137.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 15:34:18.243761 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.243695    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 15:34:18.248825 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.248807    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 15:34:18.249221 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.249209    2561 server.go:1268] "Started kubelet"
Sep 18 15:34:18.249292 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.249243    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 15:34:18.249451 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.249343    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 15:34:18.249530 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.249463    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 15:34:18.250282 ip-10-0-133-137 systemd[1]: Started Kubernetes Kubelet.
Sep 18 15:34:18.250868 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.250585    2561 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 15:34:18.250868 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.250862    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 15:34:18.253486 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.252632    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-137.ec2.internal.18d67431963d7b8e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-137.ec2.internal,UID:ip-10-0-133-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-137.ec2.internal,},FirstTimestamp:2026-09-18 15:34:18.249182094 +0000 UTC m=+1.794472758,LastTimestamp:2026-09-18 15:34:18.249182094 +0000 UTC m=+1.794472758,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-137.ec2.internal,}"
Sep 18 15:34:18.254131 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.254112    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 15:34:18.257257 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.257235    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 15:34:18.257257 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.257251    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 15:34:18.257416 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.257314    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found"
Sep 18 15:34:18.257867 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.257848    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 15:34:18.257867 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.257852    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 15:34:18.257995 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.257877    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 15:34:18.258033 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.258010    2561 factory.go:55] Registering systemd factory
Sep 18 15:34:18.258033 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.258024    2561 factory.go:223] Registration of the systemd container factory successfully
Sep 18 15:34:18.258101 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.258047    2561 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 15:34:18.258101 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.258054    2561 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 15:34:18.258248 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.258236    2561 factory.go:153] Registering CRI-O factory
Sep 18 15:34:18.258280 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.258250    2561 factory.go:223] Registration of the crio container factory successfully
Sep 18 15:34:18.258307 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.258295    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 15:34:18.258349 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.258311    2561 factory.go:103] Registering Raw factory
Sep 18 15:34:18.258349 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.258325    2561 manager.go:1196] Started watching for new ooms in manager
Sep 18 15:34:18.258836 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.258822    2561 manager.go:349] Starting recovery of all containers
Sep 18 15:34:18.260051 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.260029    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 15:34:18.260142 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.260118    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-137.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 15:34:18.269339 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.269181    2561 manager.go:354] Recovery completed
Sep 18 15:34:18.274813 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.274798    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:18.276816 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.276800    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:18.276890 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.276835    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:18.276890 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.276850    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:18.277301 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.277287    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 15:34:18.277301 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.277297    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:18.277406 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.277300    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 15:34:18.277406 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.277350    2561 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 15:34:18.278428 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.278416    2561 policy_none.go:49] "None policy: Start"
Sep 18 15:34:18.278497 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.278432    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 15:34:18.278497 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.278445    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 15:34:18.279281 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.279270    2561 policy_none.go:47] "Start"
Sep 18 15:34:18.282353 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.282335    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:18.282419 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.282368    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:18.282419 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.282380    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:18.291525 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.291462    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-137.ec2.internal.18d6743197e32d15  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-137.ec2.internal,UID:ip-10-0-133-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-137.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-137.ec2.internal,},FirstTimestamp:2026-09-18 15:34:18.276818197 +0000 UTC m=+1.822108862,LastTimestamp:2026-09-18 15:34:18.276818197 +0000 UTC m=+1.822108862,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-137.ec2.internal,}"
Sep 18 15:34:18.302422 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.302346    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-137.ec2.internal.18d6743197e38b7d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-137.ec2.internal,UID:ip-10-0-133-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-137.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-137.ec2.internal,},FirstTimestamp:2026-09-18 15:34:18.276842365 +0000 UTC m=+1.822133030,LastTimestamp:2026-09-18 15:34:18.276842365 +0000 UTC m=+1.822133030,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-137.ec2.internal,}"
Sep 18 15:34:18.303205 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.303185    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2ckmv"
Sep 18 15:34:18.311574 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.311557    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2ckmv"
Sep 18 15:34:18.312747 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.312655    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-137.ec2.internal.18d6743197e3c15f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-137.ec2.internal,UID:ip-10-0-133-137.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-137.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-137.ec2.internal,},FirstTimestamp:2026-09-18 15:34:18.276856159 +0000 UTC m=+1.822146826,LastTimestamp:2026-09-18 15:34:18.276856159 +0000 UTC m=+1.822146826,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-137.ec2.internal,}"
Sep 18 15:34:18.332469 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.320789    2561 manager.go:337] "Starting Device Plugin manager"
Sep 18 15:34:18.332469 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.320821    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 15:34:18.332469 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.320833    2561 server.go:87] "Starting device plugin registration server"
Sep 18 15:34:18.332469 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.321055    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 15:34:18.332469 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.321074    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 15:34:18.332469 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.321147    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 15:34:18.332469 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.321221    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 15:34:18.332469 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.321231    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 15:34:18.332469 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.321714    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 15:34:18.332469 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.321768    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-137.ec2.internal\" not found"
Sep 18 15:34:18.371826 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.371793    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 15:34:18.373036 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.373022    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 15:34:18.373128 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.373074    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 15:34:18.373128 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.373102    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 15:34:18.373225 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.373143    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 15:34:18.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.374782    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 15:34:18.422016 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.421997    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:18.422955 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.422937    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:18.423032 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.422965    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:18.423032 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.422975    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:18.423032 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.422996    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.432473 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.432439    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.432473 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.432457    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-137.ec2.internal\": node \"ip-10-0-133-137.ec2.internal\" not found"
Sep 18 15:34:18.460447 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.460429    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found"
Sep 18 15:34:18.473596 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.473579    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal"]
Sep 18 15:34:18.473656 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.473647    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:18.474424 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.474411    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:18.474483 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.474437    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:18.474483 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.474450    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:18.475742 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.475731    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:18.475868 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.475857    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.475904 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.475882    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:18.476581 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.476569    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:18.476643 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.476590    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:18.476643 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.476603    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:18.476643 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.476629    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:18.476758 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.476648    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:18.476758 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.476674    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:18.477979 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.477965    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.478026 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.477993    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:18.478711 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.478698    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:18.478777 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.478723    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:18.478777 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.478737    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:18.508214 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.508193    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-137.ec2.internal\" not found" node="ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.512494 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.512480    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-137.ec2.internal\" not found" node="ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.560174 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.560154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2079adb0a291ab6cf85aca3fef44d776-config\") pod \"kube-apiserver-proxy-ip-10-0-133-137.ec2.internal\" (UID: \"2079adb0a291ab6cf85aca3fef44d776\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.560262 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.560175    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8ec708c50d02cdb4a9c7bf906cf8193-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal\" (UID: \"d8ec708c50d02cdb4a9c7bf906cf8193\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.560262 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.560193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8ec708c50d02cdb4a9c7bf906cf8193-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal\" (UID: \"d8ec708c50d02cdb4a9c7bf906cf8193\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.561174 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.561161    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found"
Sep 18 15:34:18.660580 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.660553    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2079adb0a291ab6cf85aca3fef44d776-config\") pod \"kube-apiserver-proxy-ip-10-0-133-137.ec2.internal\" (UID: \"2079adb0a291ab6cf85aca3fef44d776\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.660580 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.660581    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8ec708c50d02cdb4a9c7bf906cf8193-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal\" (UID: \"d8ec708c50d02cdb4a9c7bf906cf8193\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.660768 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.660598    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8ec708c50d02cdb4a9c7bf906cf8193-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal\" (UID: \"d8ec708c50d02cdb4a9c7bf906cf8193\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.660768 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.660627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8ec708c50d02cdb4a9c7bf906cf8193-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal\" (UID: \"d8ec708c50d02cdb4a9c7bf906cf8193\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.660768 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.660683    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8ec708c50d02cdb4a9c7bf906cf8193-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal\" (UID: \"d8ec708c50d02cdb4a9c7bf906cf8193\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.660768 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.660687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2079adb0a291ab6cf85aca3fef44d776-config\") pod \"kube-apiserver-proxy-ip-10-0-133-137.ec2.internal\" (UID: \"2079adb0a291ab6cf85aca3fef44d776\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.661587 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.661573    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found"
Sep 18 15:34:18.762282 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.762220    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found"
Sep 18 15:34:18.812395 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.812365    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.816694 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:18.816507    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:18.862928 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.862900    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found"
Sep 18 15:34:18.963340 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:18.963311    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found"
Sep 18 15:34:19.063790 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:19.063723    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found"
Sep 18 15:34:19.164247 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:19.164220    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found"
Sep 18 15:34:19.202407 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.202376    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 15:34:19.202981 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.202515    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 15:34:19.264961 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:19.264935    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found"
Sep 18 15:34:19.313496 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.313460    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 15:34:19.315728 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.315654    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 15:29:18 +0000 UTC" deadline="2028-04-21 09:20:21.214187335 +0000 UTC"
Sep 18 15:34:19.315728 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.315700    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13937h46m1.898492031s"
Sep 18 15:34:19.348994 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.348972    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 15:34:19.350477 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.350461    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 15:34:19.350540 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.350493    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 15:34:19.350540 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.350507    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeHasSufficientPID"
Sep 18 15:34:19.365511 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:19.365491    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-137.ec2.internal\" not found"
Sep 18 15:34:19.366869 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.366856    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 15:34:19.424097 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:19.424071    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2079adb0a291ab6cf85aca3fef44d776.slice/crio-5c706b73d22f9cac5a6f5cab98caa0a8631cfe5af98ff2aa369407492c8900b3 WatchSource:0}: Error finding container 5c706b73d22f9cac5a6f5cab98caa0a8631cfe5af98ff2aa369407492c8900b3: Status 404 returned error can't find the container with id 5c706b73d22f9cac5a6f5cab98caa0a8631cfe5af98ff2aa369407492c8900b3
Sep 18 15:34:19.426061 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:19.425911    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ec708c50d02cdb4a9c7bf906cf8193.slice/crio-5420a95fa8981cbdc7c7e5b0e12ab2e1e0d9e51f133fc16fefa0b7fbdcc26e4d WatchSource:0}: Error finding container 5420a95fa8981cbdc7c7e5b0e12ab2e1e0d9e51f133fc16fefa0b7fbdcc26e4d: Status 404 returned error can't find the container with id 5420a95fa8981cbdc7c7e5b0e12ab2e1e0d9e51f133fc16fefa0b7fbdcc26e4d
Sep 18 15:34:19.430975 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.430959    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 15:34:19.457447 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.457429    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:19.471460 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.471445    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 15:34:19.473111 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.473099    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal"
Sep 18 15:34:19.484071 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.484059    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 15:34:19.601763 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:19.601700    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 15:34:20.227138 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.227110    2561 apiserver.go:52] "Watching apiserver"
Sep 18 15:34:20.233996 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.233872    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 15:34:20.235090 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.235065    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 15:34:20.235575 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.235551    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-wn6wq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn","openshift-dns/node-resolver-57tph","openshift-image-registry/node-ca-m7hxp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal","openshift-multus/multus-additional-cni-plugins-zwstv","openshift-multus/multus-ns5w4","openshift-network-diagnostics/network-check-target-v4ffn","kube-system/global-pull-secret-syncer-gjx2r","kube-system/konnectivity-agent-q2gdh","kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal","openshift-cluster-node-tuning-operator/tuned-vvp78","openshift-multus/network-metrics-daemon-vpmt5","openshift-network-operator/iptables-alerter-knkrl"]
Sep 18 15:34:20.237170 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.237115    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.238165 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.238059    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:20.238165 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.238140    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:20.239327 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.239310    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-57tph"
Sep 18 15:34:20.239867 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.239839    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 15:34:20.239963 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.239941    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 15:34:20.240436 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.240175    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 15:34:20.240436 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.240211    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 15:34:20.240436 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.240291    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pqbgz\""
Sep 18 15:34:20.240900 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.240796    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m7hxp"
Sep 18 15:34:20.242440 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.241696    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 15:34:20.242440 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.241918    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dmvm6\""
Sep 18 15:34:20.242440 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.242223    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 15:34:20.242440 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.242288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.243373 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.243313    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.243882 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.243710    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7w6b5\""
Sep 18 15:34:20.244013 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.243956    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 15:34:20.244314 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.244185    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 15:34:20.244314 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.244235    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 15:34:20.244795 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.244591    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tdnv7\""
Sep 18 15:34:20.244795 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.244626    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 15:34:20.245023 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.244842    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:20.245023 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.244900    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:20.245399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.245381    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 15:34:20.245645 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.245625    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 15:34:20.245865 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.245850    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 15:34:20.245924 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.245910    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 15:34:20.245981 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.245849    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 15:34:20.246569 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.246064    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 15:34:20.246569 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.246213    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 15:34:20.246569 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.246271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:20.246569 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.246334    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:20.246569 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.246416    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bfldw\""
Sep 18 15:34:20.247270 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.247219    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.248617 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.248597    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q2gdh"
Sep 18 15:34:20.249500 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.249253    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 15:34:20.249615 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.249537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 15:34:20.249928 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.249890    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dhh5k\""
Sep 18 15:34:20.249928 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.249898    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 15:34:20.250714 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.250612    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-knkrl"
Sep 18 15:34:20.251267 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.251216    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 15:34:20.252255 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.251757    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hqh4j\""
Sep 18 15:34:20.252255 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.252101    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 15:34:20.252392 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.252376    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.253131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.253114    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 15:34:20.253674 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.253640    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 15:34:20.255913 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.255517    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zq9td\""
Sep 18 15:34:20.255913 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.255534    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 15:34:20.255913 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.255772    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4t499\""
Sep 18 15:34:20.256107 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.255921    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 15:34:20.256107 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.255956    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 15:34:20.257994 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.257650    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 15:34:20.259458 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.259440    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 15:34:20.269992 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.269966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-log-socket\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.270098 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-cni-netd\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.270098 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270028    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/edc2606a-debe-4f28-93bb-b48e9562b2a3-ovnkube-config\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.270098 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhn95\" (UniqueName: \"kubernetes.io/projected/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-kube-api-access-rhn95\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.270098 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270084    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-run\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.270347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270105    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-sys\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.270347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/edc2606a-debe-4f28-93bb-b48e9562b2a3-env-overrides\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.270347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270148    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f4df3e2-04a4-4534-8c8d-aba5d753f87a-serviceca\") pod \"node-ca-m7hxp\" (UID: \"4f4df3e2-04a4-4534-8c8d-aba5d753f87a\") " pod="openshift-image-registry/node-ca-m7hxp"
Sep 18 15:34:20.270347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270169    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-run-k8s-cni-cncf-io\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.270347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-cni-bin\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.270347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/edc2606a-debe-4f28-93bb-b48e9562b2a3-ovn-node-metrics-cert\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.270347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-etc-selinux\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.270347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270277    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6knk6\" (UniqueName: \"kubernetes.io/projected/4f4df3e2-04a4-4534-8c8d-aba5d753f87a-kube-api-access-6knk6\") pod \"node-ca-m7hxp\" (UID: \"4f4df3e2-04a4-4534-8c8d-aba5d753f87a\") " pod="openshift-image-registry/node-ca-m7hxp"
Sep 18 15:34:20.270347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-dbus\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:20.270347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-kubernetes\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-tuned\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-run-netns\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270396    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-var-lib-kubelet\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-etc-openvswitch\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c509fb3-81ba-407a-bf4d-e64374627d39-iptables-alerter-script\") pod \"iptables-alerter-knkrl\" (UID: \"3c509fb3-81ba-407a-bf4d-e64374627d39\") " pod="openshift-network-operator/iptables-alerter-knkrl"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270473    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmtr8\" (UniqueName: \"kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8\") pod \"network-check-target-v4ffn\" (UID: \"541880d8-884d-4644-9c35-3485805a5425\") " pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-run-multus-certs\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-run-ovn\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgr8\" (UniqueName: \"kubernetes.io/projected/edc2606a-debe-4f28-93bb-b48e9562b2a3-kube-api-access-7mgr8\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a0c7140-64fe-4776-a159-aa50ed3f1751-cni-binary-copy\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270586    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wrzl\" (UniqueName: \"kubernetes.io/projected/bb4d3104-199c-402d-9f9e-ca63e562073c-kube-api-access-8wrzl\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270614    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wclw\" (UniqueName: \"kubernetes.io/projected/e39d26a4-b893-4a51-b493-4d1b69476b2a-kube-api-access-8wclw\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270676    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-kubelet-config\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270710    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-var-lib-cni-bin\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f4df3e2-04a4-4534-8c8d-aba5d753f87a-host\") pod \"node-ca-m7hxp\" (UID: \"4f4df3e2-04a4-4534-8c8d-aba5d753f87a\") " pod="openshift-image-registry/node-ca-m7hxp"
Sep 18 15:34:20.270772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2c97-48b2-4eb6-848d-b997c400d782-agent-certs\") pod \"konnectivity-agent-q2gdh\" (UID: \"9a0c2c97-48b2-4eb6-848d-b997c400d782\") " pod="kube-system/konnectivity-agent-q2gdh"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-sysconfig\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-cnibin\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-var-lib-cni-multus\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270892    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb4d3104-199c-402d-9f9e-ca63e562073c-cnibin\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a0c7140-64fe-4776-a159-aa50ed3f1751-multus-daemon-config\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnphp\" (UniqueName: \"kubernetes.io/projected/f2339bc3-3053-47db-b94c-dd9c926069a1-kube-api-access-gnphp\") pod \"node-resolver-57tph\" (UID: \"f2339bc3-3053-47db-b94c-dd9c926069a1\") " pod="openshift-dns/node-resolver-57tph"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.270987    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-systemd-units\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-run-systemd\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a0c2c97-48b2-4eb6-848d-b997c400d782-konnectivity-ca\") pod \"konnectivity-agent-q2gdh\" (UID: \"9a0c2c97-48b2-4eb6-848d-b997c400d782\") " pod="kube-system/konnectivity-agent-q2gdh"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-system-cni-dir\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-multus-conf-dir\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271100    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2339bc3-3053-47db-b94c-dd9c926069a1-tmp-dir\") pod \"node-resolver-57tph\" (UID: \"f2339bc3-3053-47db-b94c-dd9c926069a1\") " pod="openshift-dns/node-resolver-57tph"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271122    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271144    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb4d3104-199c-402d-9f9e-ca63e562073c-system-cni-dir\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.271490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271167    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-var-lib-kubelet\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271190    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-multus-cni-dir\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-multus-socket-dir-parent\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271263    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-etc-kubernetes\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-var-lib-openvswitch\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-run-openvswitch\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271335    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-sysctl-conf\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271359    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5872af-1276-43ee-af26-9d9c373a4abd-tmp\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-socket-dir\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271404    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-registration-dir\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c509fb3-81ba-407a-bf4d-e64374627d39-host-slash\") pod \"iptables-alerter-knkrl\" (UID: \"3c509fb3-81ba-407a-bf4d-e64374627d39\") " pod="openshift-network-operator/iptables-alerter-knkrl"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-host\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-kubelet\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271490    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-slash\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb4d3104-199c-402d-9f9e-ca63e562073c-cni-binary-copy\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-device-dir\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.272185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271557    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-sysctl-d\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-systemd\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqgqb\" (UniqueName: \"kubernetes.io/projected/8d5872af-1276-43ee-af26-9d9c373a4abd-kube-api-access-xqgqb\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb4d3104-199c-402d-9f9e-ca63e562073c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-sys-fs\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271726    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fsqx\" (UniqueName: \"kubernetes.io/projected/2a0c7140-64fe-4776-a159-aa50ed3f1751-kube-api-access-2fsqx\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2339bc3-3053-47db-b94c-dd9c926069a1-hosts-file\") pod \"node-resolver-57tph\" (UID: \"f2339bc3-3053-47db-b94c-dd9c926069a1\") " pod="openshift-dns/node-resolver-57tph"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271790    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/edc2606a-debe-4f28-93bb-b48e9562b2a3-ovnkube-script-lib\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-hostroot\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb4d3104-199c-402d-9f9e-ca63e562073c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271884    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-run-netns\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb4d3104-199c-402d-9f9e-ca63e562073c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55cn7\" (UniqueName: \"kubernetes.io/projected/3c509fb3-81ba-407a-bf4d-e64374627d39-kube-api-access-55cn7\") pod \"iptables-alerter-knkrl\" (UID: \"3c509fb3-81ba-407a-bf4d-e64374627d39\") " pod="openshift-network-operator/iptables-alerter-knkrl"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.271992    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-modprobe-d\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.273019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.272018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-lib-modules\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.273755 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.272046    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-node-log\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.275225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.275205    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 15:34:20.301239 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.301215    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gfhnx"
Sep 18 15:34:20.310962 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.310941    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gfhnx"
Sep 18 15:34:20.372941 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.372919    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-log-socket\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.373060 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.372945    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-cni-netd\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.373060 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.372967    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/edc2606a-debe-4f28-93bb-b48e9562b2a3-ovnkube-config\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.373060 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.372983    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhn95\" (UniqueName: \"kubernetes.io/projected/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-kube-api-access-rhn95\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.373060 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373004    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-run\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.373060 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-sys\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.373060 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373039    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-log-socket\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.373060 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373047    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/edc2606a-debe-4f28-93bb-b48e9562b2a3-env-overrides\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.373060 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373047    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-cni-netd\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f4df3e2-04a4-4534-8c8d-aba5d753f87a-serviceca\") pod \"node-ca-m7hxp\" (UID: \"4f4df3e2-04a4-4534-8c8d-aba5d753f87a\") " pod="openshift-image-registry/node-ca-m7hxp"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-run-k8s-cni-cncf-io\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373114    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-cni-bin\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/edc2606a-debe-4f28-93bb-b48e9562b2a3-ovn-node-metrics-cert\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-etc-selinux\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6knk6\" (UniqueName: \"kubernetes.io/projected/4f4df3e2-04a4-4534-8c8d-aba5d753f87a-kube-api-access-6knk6\") pod \"node-ca-m7hxp\" (UID: \"4f4df3e2-04a4-4534-8c8d-aba5d753f87a\") " pod="openshift-image-registry/node-ca-m7hxp"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373251    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-dbus\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373288    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-kubernetes\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-tuned\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373335    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-run-netns\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-var-lib-kubelet\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-etc-openvswitch\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373418    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c509fb3-81ba-407a-bf4d-e64374627d39-iptables-alerter-script\") pod \"iptables-alerter-knkrl\" (UID: \"3c509fb3-81ba-407a-bf4d-e64374627d39\") " pod="openshift-network-operator/iptables-alerter-knkrl"
Sep 18 15:34:20.373445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373444    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtr8\" (UniqueName: \"kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8\") pod \"network-check-target-v4ffn\" (UID: \"541880d8-884d-4644-9c35-3485805a5425\") " pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373467    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-run-multus-certs\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373493    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-run-ovn\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgr8\" (UniqueName: \"kubernetes.io/projected/edc2606a-debe-4f28-93bb-b48e9562b2a3-kube-api-access-7mgr8\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a0c7140-64fe-4776-a159-aa50ed3f1751-cni-binary-copy\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wrzl\" (UniqueName: \"kubernetes.io/projected/bb4d3104-199c-402d-9f9e-ca63e562073c-kube-api-access-8wrzl\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373605    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wclw\" (UniqueName: \"kubernetes.io/projected/e39d26a4-b893-4a51-b493-4d1b69476b2a-kube-api-access-8wclw\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-kubelet-config\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373653    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-var-lib-cni-bin\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373694    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f4df3e2-04a4-4534-8c8d-aba5d753f87a-host\") pod \"node-ca-m7hxp\" (UID: \"4f4df3e2-04a4-4534-8c8d-aba5d753f87a\") " pod="openshift-image-registry/node-ca-m7hxp"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373717    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373742    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2c97-48b2-4eb6-848d-b997c400d782-agent-certs\") pod \"konnectivity-agent-q2gdh\" (UID: \"9a0c2c97-48b2-4eb6-848d-b997c400d782\") " pod="kube-system/konnectivity-agent-q2gdh"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373763    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-var-lib-kubelet\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373772    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-sysconfig\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373818    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-run\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373827    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-sysconfig\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373847    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-cnibin\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.373977 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373873    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-var-lib-cni-multus\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373897    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb4d3104-199c-402d-9f9e-ca63e562073c-cnibin\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373921    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373945    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a0c7140-64fe-4776-a159-aa50ed3f1751-multus-daemon-config\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373971    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnphp\" (UniqueName: \"kubernetes.io/projected/f2339bc3-3053-47db-b94c-dd9c926069a1-kube-api-access-gnphp\") pod \"node-resolver-57tph\" (UID: \"f2339bc3-3053-47db-b94c-dd9c926069a1\") " pod="openshift-dns/node-resolver-57tph"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374000    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-systemd-units\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374025    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-run-systemd\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374052    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a0c2c97-48b2-4eb6-848d-b997c400d782-konnectivity-ca\") pod \"konnectivity-agent-q2gdh\" (UID: \"9a0c2c97-48b2-4eb6-848d-b997c400d782\") " pod="kube-system/konnectivity-agent-q2gdh"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374084    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-system-cni-dir\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374110    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-multus-conf-dir\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2339bc3-3053-47db-b94c-dd9c926069a1-tmp-dir\") pod \"node-resolver-57tph\" (UID: \"f2339bc3-3053-47db-b94c-dd9c926069a1\") " pod="openshift-dns/node-resolver-57tph"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374159    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb4d3104-199c-402d-9f9e-ca63e562073c-system-cni-dir\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374196    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-sys\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374211    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-var-lib-kubelet\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374242    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-multus-cni-dir\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374267    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-multus-socket-dir-parent\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.374796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374293    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-etc-kubernetes\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374320    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-var-lib-openvswitch\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374333    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3c509fb3-81ba-407a-bf4d-e64374627d39-iptables-alerter-script\") pod \"iptables-alerter-knkrl\" (UID: \"3c509fb3-81ba-407a-bf4d-e64374627d39\") " pod="openshift-network-operator/iptables-alerter-knkrl"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-run-openvswitch\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-sysctl-conf\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374526    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-sysctl-conf\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374587    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-run-multus-certs\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374598    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/edc2606a-debe-4f28-93bb-b48e9562b2a3-env-overrides\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-run-ovn\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374652    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-dbus\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.374852    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-kubernetes\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f4df3e2-04a4-4534-8c8d-aba5d753f87a-serviceca\") pod \"node-ca-m7hxp\" (UID: \"4f4df3e2-04a4-4534-8c8d-aba5d753f87a\") " pod="openshift-image-registry/node-ca-m7hxp"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375105    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375209    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-run-k8s-cni-cncf-io\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375297    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-kubelet-config\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375306    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2a0c7140-64fe-4776-a159-aa50ed3f1751-cni-binary-copy\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.373631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/edc2606a-debe-4f28-93bb-b48e9562b2a3-ovnkube-config\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375315    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-var-lib-cni-bin\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.375563 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375330    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-etc-selinux\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375364    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5872af-1276-43ee-af26-9d9c373a4abd-tmp\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f4df3e2-04a4-4534-8c8d-aba5d753f87a-host\") pod \"node-ca-m7hxp\" (UID: \"4f4df3e2-04a4-4534-8c8d-aba5d753f87a\") " pod="openshift-image-registry/node-ca-m7hxp"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375400    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-socket-dir\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.375463    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375499    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-socket-dir\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-cni-bin\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.375565    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret podName:cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea nodeName:}" failed. No retries permitted until 2026-09-18 15:34:20.875519848 +0000 UTC m=+4.420810505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret") pod "global-pull-secret-syncer-gjx2r" (UID: "cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375578    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb4d3104-199c-402d-9f9e-ca63e562073c-cnibin\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375590    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-registration-dir\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375624    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-run-netns\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c509fb3-81ba-407a-bf4d-e64374627d39-host-slash\") pod \"iptables-alerter-knkrl\" (UID: \"3c509fb3-81ba-407a-bf4d-e64374627d39\") " pod="openshift-network-operator/iptables-alerter-knkrl"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375685    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-cnibin\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375686    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-host\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-kubelet\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375728    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-host\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-slash\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.376399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375776    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb4d3104-199c-402d-9f9e-ca63e562073c-cni-binary-copy\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-device-dir\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375824    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-sysctl-d\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-systemd\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375869    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqgqb\" (UniqueName: \"kubernetes.io/projected/8d5872af-1276-43ee-af26-9d9c373a4abd-kube-api-access-xqgqb\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375894    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb4d3104-199c-402d-9f9e-ca63e562073c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375922    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-multus-conf-dir\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375939    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-sys-fs\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9a0c2c97-48b2-4eb6-848d-b997c400d782-konnectivity-ca\") pod \"konnectivity-agent-q2gdh\" (UID: \"9a0c2c97-48b2-4eb6-848d-b997c400d782\") " pod="kube-system/konnectivity-agent-q2gdh"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.375997    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-sys-fs\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376055    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-etc-openvswitch\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376202    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-run-systemd\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376203    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2339bc3-3053-47db-b94c-dd9c926069a1-tmp-dir\") pod \"node-resolver-57tph\" (UID: \"f2339bc3-3053-47db-b94c-dd9c926069a1\") " pod="openshift-dns/node-resolver-57tph"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376226    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fsqx\" (UniqueName: \"kubernetes.io/projected/2a0c7140-64fe-4776-a159-aa50ed3f1751-kube-api-access-2fsqx\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376256    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2339bc3-3053-47db-b94c-dd9c926069a1-hosts-file\") pod \"node-resolver-57tph\" (UID: \"f2339bc3-3053-47db-b94c-dd9c926069a1\") " pod="openshift-dns/node-resolver-57tph"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376283    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/edc2606a-debe-4f28-93bb-b48e9562b2a3-ovnkube-script-lib\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.377225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376295    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-multus-socket-dir-parent\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-hostroot\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb4d3104-199c-402d-9f9e-ca63e562073c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376375    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb4d3104-199c-402d-9f9e-ca63e562073c-system-cni-dir\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376389    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-run-netns\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376412    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb4d3104-199c-402d-9f9e-ca63e562073c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376418    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-var-lib-kubelet\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55cn7\" (UniqueName: \"kubernetes.io/projected/3c509fb3-81ba-407a-bf4d-e64374627d39-kube-api-access-55cn7\") pod \"iptables-alerter-knkrl\" (UID: \"3c509fb3-81ba-407a-bf4d-e64374627d39\") " pod="openshift-network-operator/iptables-alerter-knkrl"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2a0c7140-64fe-4776-a159-aa50ed3f1751-multus-daemon-config\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376465    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-multus-cni-dir\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-modprobe-d\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376501    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-lib-modules\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376550    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-system-cni-dir\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-modprobe-d\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376593    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.379006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376617    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-etc-kubernetes\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376675    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-var-lib-openvswitch\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376716    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-run-openvswitch\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376727    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb4d3104-199c-402d-9f9e-ca63e562073c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376765    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376797    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-systemd-units\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376831    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-host-var-lib-cni-multus\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376884    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-registration-dir\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.376993    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2339bc3-3053-47db-b94c-dd9c926069a1-hosts-file\") pod \"node-resolver-57tph\" (UID: \"f2339bc3-3053-47db-b94c-dd9c926069a1\") " pod="openshift-dns/node-resolver-57tph"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.377117    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb4d3104-199c-402d-9f9e-ca63e562073c-cni-binary-copy\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.377169    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-kubelet\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.377204    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-slash\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.377207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3c509fb3-81ba-407a-bf4d-e64374627d39-host-slash\") pod \"iptables-alerter-knkrl\" (UID: \"3c509fb3-81ba-407a-bf4d-e64374627d39\") " pod="openshift-network-operator/iptables-alerter-knkrl"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.377243    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-device-dir\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.377262    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-systemd\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.377276    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-node-log\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.377343    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-sysctl-d\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.377344    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-node-log\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.379792 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.377418    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:20.380625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.377424    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d5872af-1276-43ee-af26-9d9c373a4abd-lib-modules\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.380625 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.377474    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs podName:e39d26a4-b893-4a51-b493-4d1b69476b2a nodeName:}" failed. No retries permitted until 2026-09-18 15:34:20.877457891 +0000 UTC m=+4.422748544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs") pod "network-metrics-daemon-vpmt5" (UID: "e39d26a4-b893-4a51-b493-4d1b69476b2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:20.380625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.377847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bb4d3104-199c-402d-9f9e-ca63e562073c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.380625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.377897    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2a0c7140-64fe-4776-a159-aa50ed3f1751-hostroot\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.380625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.377474    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/edc2606a-debe-4f28-93bb-b48e9562b2a3-ovnkube-script-lib\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.380625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.378172    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/edc2606a-debe-4f28-93bb-b48e9562b2a3-host-run-netns\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.380625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.378365    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bb4d3104-199c-402d-9f9e-ca63e562073c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.380625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.378978    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/edc2606a-debe-4f28-93bb-b48e9562b2a3-ovn-node-metrics-cert\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.380625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.380072    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d5872af-1276-43ee-af26-9d9c373a4abd-etc-tuned\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.380625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.380089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2c97-48b2-4eb6-848d-b997c400d782-agent-certs\") pod \"konnectivity-agent-q2gdh\" (UID: \"9a0c2c97-48b2-4eb6-848d-b997c400d782\") " pod="kube-system/konnectivity-agent-q2gdh"
Sep 18 15:34:20.381111 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.380675    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d5872af-1276-43ee-af26-9d9c373a4abd-tmp\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.381818 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.381796    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:34:20.381914 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.381823    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:34:20.381914 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.381840    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nmtr8 for pod openshift-network-diagnostics/network-check-target-v4ffn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:20.381914 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.381895    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8 podName:541880d8-884d-4644-9c35-3485805a5425 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:20.881880729 +0000 UTC m=+4.427171398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nmtr8" (UniqueName: "kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8") pod "network-check-target-v4ffn" (UID: "541880d8-884d-4644-9c35-3485805a5425") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:20.382089 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.382026    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal" event={"ID":"2079adb0a291ab6cf85aca3fef44d776","Type":"ContainerStarted","Data":"5c706b73d22f9cac5a6f5cab98caa0a8631cfe5af98ff2aa369407492c8900b3"}
Sep 18 15:34:20.383596 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.383573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" event={"ID":"d8ec708c50d02cdb4a9c7bf906cf8193","Type":"ContainerStarted","Data":"5420a95fa8981cbdc7c7e5b0e12ab2e1e0d9e51f133fc16fefa0b7fbdcc26e4d"}
Sep 18 15:34:20.385159 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.384996    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgr8\" (UniqueName: \"kubernetes.io/projected/edc2606a-debe-4f28-93bb-b48e9562b2a3-kube-api-access-7mgr8\") pod \"ovnkube-node-wn6wq\" (UID: \"edc2606a-debe-4f28-93bb-b48e9562b2a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.385159 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.385132    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6knk6\" (UniqueName: \"kubernetes.io/projected/4f4df3e2-04a4-4534-8c8d-aba5d753f87a-kube-api-access-6knk6\") pod \"node-ca-m7hxp\" (UID: \"4f4df3e2-04a4-4534-8c8d-aba5d753f87a\") " pod="openshift-image-registry/node-ca-m7hxp"
Sep 18 15:34:20.385618 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.385599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhn95\" (UniqueName: \"kubernetes.io/projected/8bf3b15a-d7a0-41d3-96b7-451ad29ef767-kube-api-access-rhn95\") pod \"aws-ebs-csi-driver-node-jpndn\" (UID: \"8bf3b15a-d7a0-41d3-96b7-451ad29ef767\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.386984 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.386499    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wrzl\" (UniqueName: \"kubernetes.io/projected/bb4d3104-199c-402d-9f9e-ca63e562073c-kube-api-access-8wrzl\") pod \"multus-additional-cni-plugins-zwstv\" (UID: \"bb4d3104-199c-402d-9f9e-ca63e562073c\") " pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.390843 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.390818    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wclw\" (UniqueName: \"kubernetes.io/projected/e39d26a4-b893-4a51-b493-4d1b69476b2a-kube-api-access-8wclw\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:20.393165 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.393139    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnphp\" (UniqueName: \"kubernetes.io/projected/f2339bc3-3053-47db-b94c-dd9c926069a1-kube-api-access-gnphp\") pod \"node-resolver-57tph\" (UID: \"f2339bc3-3053-47db-b94c-dd9c926069a1\") " pod="openshift-dns/node-resolver-57tph"
Sep 18 15:34:20.393475 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.393450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fsqx\" (UniqueName: \"kubernetes.io/projected/2a0c7140-64fe-4776-a159-aa50ed3f1751-kube-api-access-2fsqx\") pod \"multus-ns5w4\" (UID: \"2a0c7140-64fe-4776-a159-aa50ed3f1751\") " pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.393757 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.393735    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55cn7\" (UniqueName: \"kubernetes.io/projected/3c509fb3-81ba-407a-bf4d-e64374627d39-kube-api-access-55cn7\") pod \"iptables-alerter-knkrl\" (UID: \"3c509fb3-81ba-407a-bf4d-e64374627d39\") " pod="openshift-network-operator/iptables-alerter-knkrl"
Sep 18 15:34:20.394045 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.394013    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqgqb\" (UniqueName: \"kubernetes.io/projected/8d5872af-1276-43ee-af26-9d9c373a4abd-kube-api-access-xqgqb\") pod \"tuned-vvp78\" (UID: \"8d5872af-1276-43ee-af26-9d9c373a4abd\") " pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.550275 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.550188    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ns5w4"
Sep 18 15:34:20.559949 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.559923    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-57tph"
Sep 18 15:34:20.569513 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.569487    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m7hxp"
Sep 18 15:34:20.581195 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.581175    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:20.588337 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.588318    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zwstv"
Sep 18 15:34:20.595816 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.595794    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn"
Sep 18 15:34:20.603274 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.603255    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q2gdh"
Sep 18 15:34:20.611611 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.611593    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-knkrl"
Sep 18 15:34:20.618187 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.618170    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vvp78"
Sep 18 15:34:20.881108 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.881018    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:20.881108 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.881087    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:20.881306 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.881167    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:20.881306 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.881198    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:20.881306 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.881237    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret podName:cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea nodeName:}" failed. No retries permitted until 2026-09-18 15:34:21.881220353 +0000 UTC m=+5.426511027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret") pod "global-pull-secret-syncer-gjx2r" (UID: "cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:20.881306 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.881263    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs podName:e39d26a4-b893-4a51-b493-4d1b69476b2a nodeName:}" failed. No retries permitted until 2026-09-18 15:34:21.881244561 +0000 UTC m=+5.426535216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs") pod "network-metrics-daemon-vpmt5" (UID: "e39d26a4-b893-4a51-b493-4d1b69476b2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:20.982017 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:20.981982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtr8\" (UniqueName: \"kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8\") pod \"network-check-target-v4ffn\" (UID: \"541880d8-884d-4644-9c35-3485805a5425\") " pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:20.982210 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.982148    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:34:20.982210 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.982166    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:34:20.982210 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.982179    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nmtr8 for pod openshift-network-diagnostics/network-check-target-v4ffn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:20.982342 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:20.982231    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8 podName:541880d8-884d-4644-9c35-3485805a5425 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:21.98221891 +0000 UTC m=+5.527509583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmtr8" (UniqueName: "kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8") pod "network-check-target-v4ffn" (UID: "541880d8-884d-4644-9c35-3485805a5425") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:21.313430 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.313341    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 15:29:20 +0000 UTC" deadline="2028-02-24 11:33:54.053456616 +0000 UTC"
Sep 18 15:34:21.313430 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.313377    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12571h59m32.74008428s"
Sep 18 15:34:21.368031 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:21.367856    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c509fb3_81ba_407a_bf4d_e64374627d39.slice/crio-5a4ca424fe8e9cb8866e3551cadcecf8ae486f030b9cc8f995824a1552854138 WatchSource:0}: Error finding container 5a4ca424fe8e9cb8866e3551cadcecf8ae486f030b9cc8f995824a1552854138: Status 404 returned error can't find the container with id 5a4ca424fe8e9cb8866e3551cadcecf8ae486f030b9cc8f995824a1552854138
Sep 18 15:34:21.369370 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:21.369331    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb4d3104_199c_402d_9f9e_ca63e562073c.slice/crio-e68a66869555a85284b24956eeffa1f4c8504f3e61b65d3b3d28fc9b68709893 WatchSource:0}: Error finding container e68a66869555a85284b24956eeffa1f4c8504f3e61b65d3b3d28fc9b68709893: Status 404 returned error can't find the container with id e68a66869555a85284b24956eeffa1f4c8504f3e61b65d3b3d28fc9b68709893
Sep 18 15:34:21.371171 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:21.371138    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc2606a_debe_4f28_93bb_b48e9562b2a3.slice/crio-9a94458861d691ec4f0d68c49dfe4a70e18f0949e7b3cd17f02b83a34075a241 WatchSource:0}: Error finding container 9a94458861d691ec4f0d68c49dfe4a70e18f0949e7b3cd17f02b83a34075a241: Status 404 returned error can't find the container with id 9a94458861d691ec4f0d68c49dfe4a70e18f0949e7b3cd17f02b83a34075a241
Sep 18 15:34:21.373621 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.373378    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:21.373621 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:21.373511    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:21.375016 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:21.374974    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf3b15a_d7a0_41d3_96b7_451ad29ef767.slice/crio-281c7166627cb4ec42448eda0ff65bf72fb8462370d9f1ee71a2cc45fef7bf93 WatchSource:0}: Error finding container 281c7166627cb4ec42448eda0ff65bf72fb8462370d9f1ee71a2cc45fef7bf93: Status 404 returned error can't find the container with id 281c7166627cb4ec42448eda0ff65bf72fb8462370d9f1ee71a2cc45fef7bf93
Sep 18 15:34:21.375990 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:21.375872    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4df3e2_04a4_4534_8c8d_aba5d753f87a.slice/crio-4d7caca50ad18c06ff20608c82afdbd3f65cf21289b447babe86620a22197b35 WatchSource:0}: Error finding container 4d7caca50ad18c06ff20608c82afdbd3f65cf21289b447babe86620a22197b35: Status 404 returned error can't find the container with id 4d7caca50ad18c06ff20608c82afdbd3f65cf21289b447babe86620a22197b35
Sep 18 15:34:21.377379 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:21.377129    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2339bc3_3053_47db_b94c_dd9c926069a1.slice/crio-a5d948c8c379a79f198f147dea7c0f5811e61b8df653aeacfaf9b5e9c954a56e WatchSource:0}: Error finding container a5d948c8c379a79f198f147dea7c0f5811e61b8df653aeacfaf9b5e9c954a56e: Status 404 returned error can't find the container with id a5d948c8c379a79f198f147dea7c0f5811e61b8df653aeacfaf9b5e9c954a56e
Sep 18 15:34:21.377535 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:21.377473    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0c2c97_48b2_4eb6_848d_b997c400d782.slice/crio-eed92a6afaaa7eddebc7157aba51d0d05737be944a77ce7f3c12df3afbce1136 WatchSource:0}: Error finding container eed92a6afaaa7eddebc7157aba51d0d05737be944a77ce7f3c12df3afbce1136: Status 404 returned error can't find the container with id eed92a6afaaa7eddebc7157aba51d0d05737be944a77ce7f3c12df3afbce1136
Sep 18 15:34:21.387517 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.387487    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zwstv" event={"ID":"bb4d3104-199c-402d-9f9e-ca63e562073c","Type":"ContainerStarted","Data":"e68a66869555a85284b24956eeffa1f4c8504f3e61b65d3b3d28fc9b68709893"}
Sep 18 15:34:21.389111 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.389085    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-knkrl" event={"ID":"3c509fb3-81ba-407a-bf4d-e64374627d39","Type":"ContainerStarted","Data":"5a4ca424fe8e9cb8866e3551cadcecf8ae486f030b9cc8f995824a1552854138"}
Sep 18 15:34:21.390228 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.390196    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ns5w4" event={"ID":"2a0c7140-64fe-4776-a159-aa50ed3f1751","Type":"ContainerStarted","Data":"ff46e94879fb73441d36cd02477298bb13363eadb33d70274fd14c0656bd38db"}
Sep 18 15:34:21.391604 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.391583    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vvp78" event={"ID":"8d5872af-1276-43ee-af26-9d9c373a4abd","Type":"ContainerStarted","Data":"ce5ae0365ddd507449debdf385e393efb556e7735a4355220d9366b892f68d76"}
Sep 18 15:34:21.392976 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.392952    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-57tph" event={"ID":"f2339bc3-3053-47db-b94c-dd9c926069a1","Type":"ContainerStarted","Data":"a5d948c8c379a79f198f147dea7c0f5811e61b8df653aeacfaf9b5e9c954a56e"}
Sep 18 15:34:21.394318 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.394270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m7hxp" event={"ID":"4f4df3e2-04a4-4534-8c8d-aba5d753f87a","Type":"ContainerStarted","Data":"4d7caca50ad18c06ff20608c82afdbd3f65cf21289b447babe86620a22197b35"}
Sep 18 15:34:21.395992 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.395966    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq" event={"ID":"edc2606a-debe-4f28-93bb-b48e9562b2a3","Type":"ContainerStarted","Data":"9a94458861d691ec4f0d68c49dfe4a70e18f0949e7b3cd17f02b83a34075a241"}
Sep 18 15:34:21.397637 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.397612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q2gdh" event={"ID":"9a0c2c97-48b2-4eb6-848d-b997c400d782","Type":"ContainerStarted","Data":"eed92a6afaaa7eddebc7157aba51d0d05737be944a77ce7f3c12df3afbce1136"}
Sep 18 15:34:21.398974 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.398956    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn" event={"ID":"8bf3b15a-d7a0-41d3-96b7-451ad29ef767","Type":"ContainerStarted","Data":"281c7166627cb4ec42448eda0ff65bf72fb8462370d9f1ee71a2cc45fef7bf93"}
Sep 18 15:34:21.887842 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.887522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:21.887842 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:21.887793    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:21.888061 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.887858    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:21.888061 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:21.887869    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret podName:cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea nodeName:}" failed. No retries permitted until 2026-09-18 15:34:23.887848984 +0000 UTC m=+7.433139653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret") pod "global-pull-secret-syncer-gjx2r" (UID: "cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:21.888061 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:21.887981    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:21.888061 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:21.888037    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs podName:e39d26a4-b893-4a51-b493-4d1b69476b2a nodeName:}" failed. No retries permitted until 2026-09-18 15:34:23.888020055 +0000 UTC m=+7.433310718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs") pod "network-metrics-daemon-vpmt5" (UID: "e39d26a4-b893-4a51-b493-4d1b69476b2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:21.989132 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:21.989096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtr8\" (UniqueName: \"kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8\") pod \"network-check-target-v4ffn\" (UID: \"541880d8-884d-4644-9c35-3485805a5425\") " pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:21.989299 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:21.989246    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:34:21.989299 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:21.989267    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:34:21.989299 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:21.989279    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nmtr8 for pod openshift-network-diagnostics/network-check-target-v4ffn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:21.989453 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:21.989336    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8 podName:541880d8-884d-4644-9c35-3485805a5425 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:23.98931698 +0000 UTC m=+7.534607654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmtr8" (UniqueName: "kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8") pod "network-check-target-v4ffn" (UID: "541880d8-884d-4644-9c35-3485805a5425") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:22.314548 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:22.314432    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 15:29:20 +0000 UTC" deadline="2028-06-04 05:57:24.425715748 +0000 UTC"
Sep 18 15:34:22.314548 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:22.314497    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14990h23m2.111222542s"
Sep 18 15:34:22.374806 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:22.374774    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:22.374975 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:22.374904    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:22.375426 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:22.375239    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:22.375426 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:22.375339    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:22.412203 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:22.411810    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal" event={"ID":"2079adb0a291ab6cf85aca3fef44d776","Type":"ContainerStarted","Data":"515fa24631ad3e0e61e2aaf7db550b9db8feedec98f8b904075f22bf99c9be98"}
Sep 18 15:34:22.424537 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:22.424401    2561 generic.go:356] "Generic (PLEG): container finished" podID="d8ec708c50d02cdb4a9c7bf906cf8193" containerID="cd40c18c40fd794bfb61abfa21a0f8b5d8e25d76b53091ecc909ad52a2e99cd5" exitCode=0
Sep 18 15:34:22.424537 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:22.424447    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" event={"ID":"d8ec708c50d02cdb4a9c7bf906cf8193","Type":"ContainerDied","Data":"cd40c18c40fd794bfb61abfa21a0f8b5d8e25d76b53091ecc909ad52a2e99cd5"}
Sep 18 15:34:22.475823 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:22.475770    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-137.ec2.internal" podStartSLOduration=3.475750798 podStartE2EDuration="3.475750798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:22.445056217 +0000 UTC m=+5.990346887" watchObservedRunningTime="2026-09-18 15:34:22.475750798 +0000 UTC m=+6.021041474"
Sep 18 15:34:23.374543 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:23.374027    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:23.374543 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:23.374184    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:23.441995 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:23.441911    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" event={"ID":"d8ec708c50d02cdb4a9c7bf906cf8193","Type":"ContainerStarted","Data":"50e5b0f73e07e4ef38ba0b6f06bf63b006edfe44526ace112ab51a07e762b195"}
Sep 18 15:34:23.901960 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:23.901929    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:23.902175 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:23.901981    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:23.902175 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:23.902097    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:23.902175 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:23.902145    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs podName:e39d26a4-b893-4a51-b493-4d1b69476b2a nodeName:}" failed. No retries permitted until 2026-09-18 15:34:27.902133094 +0000 UTC m=+11.447423751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs") pod "network-metrics-daemon-vpmt5" (UID: "e39d26a4-b893-4a51-b493-4d1b69476b2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:23.902494 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:23.902475    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:23.902591 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:23.902528    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret podName:cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea nodeName:}" failed. No retries permitted until 2026-09-18 15:34:27.902513988 +0000 UTC m=+11.447804646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret") pod "global-pull-secret-syncer-gjx2r" (UID: "cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:24.002472 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:24.002438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtr8\" (UniqueName: \"kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8\") pod \"network-check-target-v4ffn\" (UID: \"541880d8-884d-4644-9c35-3485805a5425\") " pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:24.002701 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:24.002683    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:34:24.002783 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:24.002709    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:34:24.002783 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:24.002722    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nmtr8 for pod openshift-network-diagnostics/network-check-target-v4ffn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:24.002783 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:24.002778    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8 podName:541880d8-884d-4644-9c35-3485805a5425 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:28.002759951 +0000 UTC m=+11.548050606 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmtr8" (UniqueName: "kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8") pod "network-check-target-v4ffn" (UID: "541880d8-884d-4644-9c35-3485805a5425") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:24.376015 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:24.375934    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:24.376464 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:24.376062    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:24.376464 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:24.376422    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:24.376583 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:24.376512    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:25.374155 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:25.374115    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:25.374315 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:25.374264    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:26.374227 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:26.374192    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:26.374688 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:26.374312    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:26.374758 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:26.374739    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:26.374870 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:26.374842    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:27.373934 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:27.373893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:27.374125 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:27.374045    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:27.936195 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:27.936150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:27.936654 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:27.936236    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:27.936654 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:27.936377    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:27.936654 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:27.936408    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:27.936654 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:27.936438    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs podName:e39d26a4-b893-4a51-b493-4d1b69476b2a nodeName:}" failed. No retries permitted until 2026-09-18 15:34:35.936420124 +0000 UTC m=+19.481710800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs") pod "network-metrics-daemon-vpmt5" (UID: "e39d26a4-b893-4a51-b493-4d1b69476b2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:27.936654 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:27.936457    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret podName:cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea nodeName:}" failed. No retries permitted until 2026-09-18 15:34:35.936447483 +0000 UTC m=+19.481738134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret") pod "global-pull-secret-syncer-gjx2r" (UID: "cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:28.036722 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:28.036688    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtr8\" (UniqueName: \"kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8\") pod \"network-check-target-v4ffn\" (UID: \"541880d8-884d-4644-9c35-3485805a5425\") " pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:28.036894 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:28.036856    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:34:28.036894 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:28.036878    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:34:28.036894 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:28.036888    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nmtr8 for pod openshift-network-diagnostics/network-check-target-v4ffn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:28.037053 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:28.036951    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8 podName:541880d8-884d-4644-9c35-3485805a5425 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:36.036932396 +0000 UTC m=+19.582223066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmtr8" (UniqueName: "kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8") pod "network-check-target-v4ffn" (UID: "541880d8-884d-4644-9c35-3485805a5425") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:28.375256 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:28.374803    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:28.375256 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:28.374899    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:28.375256 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:28.375158    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:28.375503 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:28.375268    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:29.373925 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:29.373817    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:29.374341 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:29.373967    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:30.374032 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:30.373996    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:30.374432 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:30.374130    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:30.374488 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:30.374469    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:30.374656 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:30.374585    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:31.374072 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:31.374037    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:31.374473 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:31.374168    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:32.373941 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:32.373909    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:32.374154 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:32.373909    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:32.374154 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:32.374041    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:32.374154 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:32.374107    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:33.373803 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:33.373771    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:33.373964 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:33.373908    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:34.373589 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:34.373555    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:34.374048 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:34.373678    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:34.374048 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:34.373704    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:34.374048 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:34.373795    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:35.373324 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:35.373286    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:35.373648 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:35.373420    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:35.994975 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:35.994941    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:35.995164 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:35.995104    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:35.995234 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:35.995109    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:35.995234 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:35.995173    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs podName:e39d26a4-b893-4a51-b493-4d1b69476b2a nodeName:}" failed. No retries permitted until 2026-09-18 15:34:51.995152066 +0000 UTC m=+35.540442732 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs") pod "network-metrics-daemon-vpmt5" (UID: "e39d26a4-b893-4a51-b493-4d1b69476b2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:35.995234 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:35.995193    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:35.995380 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:35.995252    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret podName:cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea nodeName:}" failed. No retries permitted until 2026-09-18 15:34:51.99523559 +0000 UTC m=+35.540526245 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret") pod "global-pull-secret-syncer-gjx2r" (UID: "cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:36.095951 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:36.095923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtr8\" (UniqueName: \"kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8\") pod \"network-check-target-v4ffn\" (UID: \"541880d8-884d-4644-9c35-3485805a5425\") " pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:36.096138 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:36.096071    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:34:36.096138 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:36.096090    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:34:36.096138 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:36.096101    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nmtr8 for pod openshift-network-diagnostics/network-check-target-v4ffn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:36.096279 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:36.096161    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8 podName:541880d8-884d-4644-9c35-3485805a5425 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:52.096143513 +0000 UTC m=+35.641434168 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmtr8" (UniqueName: "kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8") pod "network-check-target-v4ffn" (UID: "541880d8-884d-4644-9c35-3485805a5425") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:36.373805 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:36.373703    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:36.373805 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:36.373735    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:36.374320 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:36.373843    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:36.374320 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:36.374002    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:37.373638 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:37.373598    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:37.373834 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:37.373756    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:38.376768 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:38.376731    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:38.377255 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:38.377014    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:38.377255 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:38.377005    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:38.377448 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:38.377425    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:39.374345 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.373897    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:39.374532 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:39.374461    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:39.473525 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.473493    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ns5w4" event={"ID":"2a0c7140-64fe-4776-a159-aa50ed3f1751","Type":"ContainerStarted","Data":"fb24a89c067b530f37103c0b89e63f5d2d3ed832544f5a78e1e648262ec22092"}
Sep 18 15:34:39.475034 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.475012    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vvp78" event={"ID":"8d5872af-1276-43ee-af26-9d9c373a4abd","Type":"ContainerStarted","Data":"d337a742d89cddfac34a6da8a965f5d2a1ecfa22ad4eaa5cecb6da33f0e289d3"}
Sep 18 15:34:39.478986 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.478964    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-57tph" event={"ID":"f2339bc3-3053-47db-b94c-dd9c926069a1","Type":"ContainerStarted","Data":"cd394bc9e785966789dfabb6bc8b80ca90157985d455b0a9716bb7d9dc667605"}
Sep 18 15:34:39.480565 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.480539    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m7hxp" event={"ID":"4f4df3e2-04a4-4534-8c8d-aba5d753f87a","Type":"ContainerStarted","Data":"e12d7a4d94cbecaa103bd672111a715baf53b11487e66ca5013be17e00a73118"}
Sep 18 15:34:39.483482 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.483462    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq" event={"ID":"edc2606a-debe-4f28-93bb-b48e9562b2a3","Type":"ContainerStarted","Data":"09ee6b9ca75b0336d41568c274066b3b49aecb4603090007d67416c68e002d47"}
Sep 18 15:34:39.483579 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.483525    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq" event={"ID":"edc2606a-debe-4f28-93bb-b48e9562b2a3","Type":"ContainerStarted","Data":"ef24889961f500bd6fb9701d19f31a2c2fde7c94ef019931bd485de26c3d894e"}
Sep 18 15:34:39.483579 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.483573    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq" event={"ID":"edc2606a-debe-4f28-93bb-b48e9562b2a3","Type":"ContainerStarted","Data":"ea981b603ad1e9ee7a6383e546c1edb3f6eaeba90589541a936272b3840ea02f"}
Sep 18 15:34:39.483694 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.483603    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq" event={"ID":"edc2606a-debe-4f28-93bb-b48e9562b2a3","Type":"ContainerStarted","Data":"d1cf66a220ddc8551e318de3b154002926cf6d009173467761b7d13f676850a0"}
Sep 18 15:34:39.483694 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.483642    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq" event={"ID":"edc2606a-debe-4f28-93bb-b48e9562b2a3","Type":"ContainerStarted","Data":"9ab911362148d5dc136abe118663b3a0ddec57e8ebb5d0949b98fdc3b904160b"}
Sep 18 15:34:39.483772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.483700    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq" event={"ID":"edc2606a-debe-4f28-93bb-b48e9562b2a3","Type":"ContainerStarted","Data":"fd10e6fc7bc3921ed1f08a90b5dc5fdf906f23b126334d44b3610c7e7dcd7a65"}
Sep 18 15:34:39.484803 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.484784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q2gdh" event={"ID":"9a0c2c97-48b2-4eb6-848d-b997c400d782","Type":"ContainerStarted","Data":"1806342c2d6236ec75729aa32427f3f6e3ec8295940ada4b982e099174badff2"}
Sep 18 15:34:39.486129 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.486110    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn" event={"ID":"8bf3b15a-d7a0-41d3-96b7-451ad29ef767","Type":"ContainerStarted","Data":"576ccc9ba56b4afe80e53aa26f20879f98f07383ee9d068944eca0388a533dec"}
Sep 18 15:34:39.487523 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.487505    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb4d3104-199c-402d-9f9e-ca63e562073c" containerID="b94f34b6e674ae8fbe209d46d1701e5c33227d371faffec68a74f45fab980740" exitCode=0
Sep 18 15:34:39.487605 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.487532    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zwstv" event={"ID":"bb4d3104-199c-402d-9f9e-ca63e562073c","Type":"ContainerDied","Data":"b94f34b6e674ae8fbe209d46d1701e5c33227d371faffec68a74f45fab980740"}
Sep 18 15:34:39.503583 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.503549    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-137.ec2.internal" podStartSLOduration=20.503538755 podStartE2EDuration="20.503538755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:23.495196383 +0000 UTC m=+7.040487060" watchObservedRunningTime="2026-09-18 15:34:39.503538755 +0000 UTC m=+23.048829428"
Sep 18 15:34:39.503708 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.503687    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ns5w4" podStartSLOduration=4.25375503 podStartE2EDuration="21.503680685s" totalImagesPullingTime="17.249925655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:39.502537201 +0000 UTC m=+23.047827875" watchObservedRunningTime="2026-09-18 15:34:39.503680685 +0000 UTC m=+23.048971358"
Sep 18 15:34:39.610561 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.610475    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vvp78" podStartSLOduration=4.394740348 podStartE2EDuration="21.610458281s" totalImagesPullingTime="17.215717933s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:39.587992619 +0000 UTC m=+23.133283294" watchObservedRunningTime="2026-09-18 15:34:39.610458281 +0000 UTC m=+23.155749012"
Sep 18 15:34:39.655720 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.655653    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m7hxp" podStartSLOduration=4.438871149 podStartE2EDuration="21.655633027s" totalImagesPullingTime="17.216761878s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:39.610300609 +0000 UTC m=+23.155591285" watchObservedRunningTime="2026-09-18 15:34:39.655633027 +0000 UTC m=+23.200923702"
Sep 18 15:34:39.656137 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.656108    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q2gdh" podStartSLOduration=8.505665225 podStartE2EDuration="21.656097966s" totalImagesPullingTime="13.150432741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:39.654107419 +0000 UTC m=+23.199398092" watchObservedRunningTime="2026-09-18 15:34:39.656097966 +0000 UTC m=+23.201388640"
Sep 18 15:34:39.729030 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.728972    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-57tph" podStartSLOduration=4.513350711 podStartE2EDuration="21.728956876s" totalImagesPullingTime="17.215606165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:39.722614664 +0000 UTC m=+23.267905338" watchObservedRunningTime="2026-09-18 15:34:39.728956876 +0000 UTC m=+23.274247549"
Sep 18 15:34:39.856747 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:39.856710    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 15:34:40.331861 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:40.331717    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T15:34:39.856735355Z","UUID":"60d279b2-22e7-4c6d-8b0b-57755cb280e2","Handler":null,"Name":"","Endpoint":""}
Sep 18 15:34:40.333579 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:40.333509    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 15:34:40.333579 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:40.333543    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 15:34:40.374226 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:40.374197    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:40.374355 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:40.374230    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:40.374355 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:40.374303    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:40.374434 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:40.374417    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:40.492844 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:40.492751    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn" event={"ID":"8bf3b15a-d7a0-41d3-96b7-451ad29ef767","Type":"ContainerStarted","Data":"fa319b2a9002ade575cd62aaa04c01b2ad89afe3662366d8110bbab086f87229"}
Sep 18 15:34:40.494573 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:40.494542    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-knkrl" event={"ID":"3c509fb3-81ba-407a-bf4d-e64374627d39","Type":"ContainerStarted","Data":"40e88353966913297c341c13520a273a5b872c74db0ebffedbf3e0447f98ffc9"}
Sep 18 15:34:41.374144 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:41.374115    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:41.374297 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:41.374218    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:41.499741 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:41.499705    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn" event={"ID":"8bf3b15a-d7a0-41d3-96b7-451ad29ef767","Type":"ContainerStarted","Data":"e0ffa424cbd4a1a997cc56ff7618c9e5dd70800c1d429833b35ac3f8a728db6a"}
Sep 18 15:34:41.503995 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:41.503960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq" event={"ID":"edc2606a-debe-4f28-93bb-b48e9562b2a3","Type":"ContainerStarted","Data":"d2c96b5c3c2486d7800c28ce1ad67328690db804cc2393c513d1066ab7ede66a"}
Sep 18 15:34:41.520623 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:41.520576    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-knkrl" podStartSLOduration=6.29619978 podStartE2EDuration="23.520562824s" totalImagesPullingTime="17.224363044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:40.51954372 +0000 UTC m=+24.064834398" watchObservedRunningTime="2026-09-18 15:34:41.520562824 +0000 UTC m=+25.065853498"
Sep 18 15:34:41.520889 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:41.520865    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jpndn" podStartSLOduration=4.400846525 podStartE2EDuration="23.520855112s" totalImagesPullingTime="19.120008587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:41.520300369 +0000 UTC m=+25.065591045" watchObservedRunningTime="2026-09-18 15:34:41.520855112 +0000 UTC m=+25.066145785"
Sep 18 15:34:42.373946 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:42.373913    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:42.374148 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:42.373922    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:42.374148 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:42.374052    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:42.374285 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:42.374145    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:43.374173 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:43.374136    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:43.374578 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:43.374257    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:43.506975 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:43.506697    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q2gdh"
Sep 18 15:34:43.507307 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:43.507261    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q2gdh"
Sep 18 15:34:44.373618 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:44.373587    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:44.373618 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:44.373629    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:44.373882 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:44.373732    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:44.373882 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:44.373837    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:44.514087 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:44.513996    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq" event={"ID":"edc2606a-debe-4f28-93bb-b48e9562b2a3","Type":"ContainerStarted","Data":"887300877afbdb1c3dca8b73412f442642da0f4932a65f7b635fc422282bf986"}
Sep 18 15:34:44.514905 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:44.514280    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:44.515957 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:44.515933    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb4d3104-199c-402d-9f9e-ca63e562073c" containerID="184b392435736fad3a50e9d9ef8524d57d9960fab2c604cf1de1885abd4dd8fa" exitCode=0
Sep 18 15:34:44.516063 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:44.515979    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zwstv" event={"ID":"bb4d3104-199c-402d-9f9e-ca63e562073c","Type":"ContainerDied","Data":"184b392435736fad3a50e9d9ef8524d57d9960fab2c604cf1de1885abd4dd8fa"}
Sep 18 15:34:44.516218 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:44.516197    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q2gdh"
Sep 18 15:34:44.516773 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:44.516700    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q2gdh"
Sep 18 15:34:44.529753 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:44.529733    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:44.567394 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:44.567356    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq" podStartSLOduration=9.324461922 podStartE2EDuration="26.567338505s" totalImagesPullingTime="17.242876583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:44.56668788 +0000 UTC m=+28.111978555" watchObservedRunningTime="2026-09-18 15:34:44.567338505 +0000 UTC m=+28.112629173"
Sep 18 15:34:44.734182 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:44.734010    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:45.374387 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:45.374353    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:45.374539 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:45.374476    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:45.518554 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:45.518523    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:45.532840 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:45.532817    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:34:46.163896 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:46.163774    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vpmt5"]
Sep 18 15:34:46.164083 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:46.163951    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:46.164083 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:46.164064    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:46.164460 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:46.164438    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v4ffn"]
Sep 18 15:34:46.164559 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:46.164543    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:46.164672 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:46.164638    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:46.171463 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:46.171441    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gjx2r"]
Sep 18 15:34:46.171545 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:46.171529    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:46.171612 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:46.171596    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:46.522238 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:46.522208    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb4d3104-199c-402d-9f9e-ca63e562073c" containerID="0ab1f75b3817f5ba3ed975db4498f43861bb2c2edb8669868e356c8ed8cfe374" exitCode=0
Sep 18 15:34:46.522605 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:46.522297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zwstv" event={"ID":"bb4d3104-199c-402d-9f9e-ca63e562073c","Type":"ContainerDied","Data":"0ab1f75b3817f5ba3ed975db4498f43861bb2c2edb8669868e356c8ed8cfe374"}
Sep 18 15:34:47.373869 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:47.373842    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:47.374003 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:47.373830    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:47.374003 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:47.373973    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:47.374084 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:47.374008    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:48.376332 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:48.376266    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:48.376704 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:48.376349    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:48.529651 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:48.529618    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb4d3104-199c-402d-9f9e-ca63e562073c" containerID="c984e4bc661696fc4598f56aa4b926fbd00bace8c38143caab9f0d32fd819e38" exitCode=0
Sep 18 15:34:48.529818 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:48.529686    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zwstv" event={"ID":"bb4d3104-199c-402d-9f9e-ca63e562073c","Type":"ContainerDied","Data":"c984e4bc661696fc4598f56aa4b926fbd00bace8c38143caab9f0d32fd819e38"}
Sep 18 15:34:49.374021 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:49.373988    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:49.374021 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:49.374015    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:49.374263 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:49.374120    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:49.374315 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:49.374253    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:50.378191 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:50.378163    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:50.378577 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:50.378291    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v4ffn" podUID="541880d8-884d-4644-9c35-3485805a5425"
Sep 18 15:34:51.373998 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.373966    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:51.374162 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:51.374072    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjx2r" podUID="cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea"
Sep 18 15:34:51.374212 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.374155    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:51.374276 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:51.374258    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vpmt5" podUID="e39d26a4-b893-4a51-b493-4d1b69476b2a"
Sep 18 15:34:51.520677 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.520634    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-137.ec2.internal" event="NodeReady"
Sep 18 15:34:51.521119 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.520785    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 15:34:51.585328 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.585294    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"]
Sep 18 15:34:51.605605 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.605582    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.609975 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.609951    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 15:34:51.610245 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.610221    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7tpfr\""
Sep 18 15:34:51.610351 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.610222    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 15:34:51.610419 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.610381    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 15:34:51.619051 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.619030    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 15:34:51.624422 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.624397    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"]
Sep 18 15:34:51.633928 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.633886    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mmqzn"]
Sep 18 15:34:51.648261 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.648232    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rqkgz"]
Sep 18 15:34:51.648370 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.648354    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:51.656946 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.656924    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 15:34:51.657160 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.657147    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9wls\""
Sep 18 15:34:51.657343 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.657332    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 15:34:51.660512 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.660492    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rqkgz"]
Sep 18 15:34:51.660609 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.660522    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mmqzn"]
Sep 18 15:34:51.660686 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.660609    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rqkgz"
Sep 18 15:34:51.665670 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.665634    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 15:34:51.666095 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.666067    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 15:34:51.666171 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.666105    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mtwcv\""
Sep 18 15:34:51.666171 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.666148    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 15:34:51.708717 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.708692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:51.708833 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.708724    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-image-registry-private-configuration\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.708908 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.708829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-certificates\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.708908 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.708865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkh5m\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-kube-api-access-zkh5m\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.709012 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.708910    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85kl2\" (UniqueName: \"kubernetes.io/projected/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-kube-api-access-85kl2\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:51.709012 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.708926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-ca-trust-extracted\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.709012 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.708980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-tmp-dir\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:51.709158 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.709019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-installation-pull-secrets\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.709158 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.709057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.709158 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.709093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-trusted-ca\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.709158 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.709131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-config-volume\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:51.709158 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.709156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-bound-sa-token\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.810267 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.810201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-config-volume\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:51.810267 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.810232    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-bound-sa-token\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.810446 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.810268    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:51.810446 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:51.810364    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 15:34:51.810446 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:51.810416    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls podName:ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:52.310398757 +0000 UTC m=+35.855689409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls") pod "dns-default-mmqzn" (UID: "ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407") : secret "dns-default-metrics-tls" not found
Sep 18 15:34:51.810565 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.810446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-image-registry-private-configuration\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.810565 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.810493    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-certificates\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.810565 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.810522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkh5m\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-kube-api-access-zkh5m\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.810751 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.810570    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85kl2\" (UniqueName: \"kubernetes.io/projected/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-kube-api-access-85kl2\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:51.810751 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.810596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-ca-trust-extracted\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.810751 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.810626    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-tmp-dir\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:51.810751 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.810658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-installation-pull-secrets\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.810751 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.810711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1-cert\") pod \"ingress-canary-rqkgz\" (UID: \"84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1\") " pod="openshift-ingress-canary/ingress-canary-rqkgz"
Sep 18 15:34:51.810751 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.810746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.811494 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.811053    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-trusted-ca\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.811494 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.811093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtplz\" (UniqueName: \"kubernetes.io/projected/84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1-kube-api-access-xtplz\") pod \"ingress-canary-rqkgz\" (UID: \"84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1\") " pod="openshift-ingress-canary/ingress-canary-rqkgz"
Sep 18 15:34:51.811494 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.811182    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-config-volume\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:51.811494 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.811356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-tmp-dir\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:51.811494 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.811423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-certificates\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.811494 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.811433    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-ca-trust-extracted\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.812006 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:51.811988    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 15:34:51.812059 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:51.812008    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbbbbbbb5-sqwnt: secret "image-registry-tls" not found
Sep 18 15:34:51.812102 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:51.812065    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls podName:b215b96a-f47b-4e8c-8df3-3c1aa336a57e nodeName:}" failed. No retries permitted until 2026-09-18 15:34:52.312048694 +0000 UTC m=+35.857339359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls") pod "image-registry-cbbbbbbb5-sqwnt" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e") : secret "image-registry-tls" not found
Sep 18 15:34:51.813966 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.813943    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-trusted-ca\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.816484 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.816464    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-image-registry-private-configuration\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.817108 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.817092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-installation-pull-secrets\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.822678 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.822636    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-bound-sa-token\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.823185 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.823169    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkh5m\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-kube-api-access-zkh5m\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:51.832556 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.832538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85kl2\" (UniqueName: \"kubernetes.io/projected/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-kube-api-access-85kl2\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:51.911674 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.911632    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtplz\" (UniqueName: \"kubernetes.io/projected/84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1-kube-api-access-xtplz\") pod \"ingress-canary-rqkgz\" (UID: \"84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1\") " pod="openshift-ingress-canary/ingress-canary-rqkgz"
Sep 18 15:34:51.911859 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.911740    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1-cert\") pod \"ingress-canary-rqkgz\" (UID: \"84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1\") " pod="openshift-ingress-canary/ingress-canary-rqkgz"
Sep 18 15:34:51.914188 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.914164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1-cert\") pod \"ingress-canary-rqkgz\" (UID: \"84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1\") " pod="openshift-ingress-canary/ingress-canary-rqkgz"
Sep 18 15:34:51.923001 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.922979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtplz\" (UniqueName: \"kubernetes.io/projected/84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1-kube-api-access-xtplz\") pod \"ingress-canary-rqkgz\" (UID: \"84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1\") " pod="openshift-ingress-canary/ingress-canary-rqkgz"
Sep 18 15:34:51.968985 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:51.968955    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rqkgz"
Sep 18 15:34:52.012322 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:52.012288    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:52.012433 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:52.012369    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:52.012492 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:52.012429    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:52.012492 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:52.012486    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:52.012579 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:52.012495    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret podName:cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea nodeName:}" failed. No retries permitted until 2026-09-18 15:35:24.01247587 +0000 UTC m=+67.557766546 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret") pod "global-pull-secret-syncer-gjx2r" (UID: "cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea") : object "kube-system"/"original-pull-secret" not registered
Sep 18 15:34:52.012579 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:52.012537    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs podName:e39d26a4-b893-4a51-b493-4d1b69476b2a nodeName:}" failed. No retries permitted until 2026-09-18 15:35:24.012520537 +0000 UTC m=+67.557811199 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs") pod "network-metrics-daemon-vpmt5" (UID: "e39d26a4-b893-4a51-b493-4d1b69476b2a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 15:34:52.113805 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:52.113724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtr8\" (UniqueName: \"kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8\") pod \"network-check-target-v4ffn\" (UID: \"541880d8-884d-4644-9c35-3485805a5425\") " pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:52.113940 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:52.113890    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 15:34:52.113940 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:52.113918    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 15:34:52.113940 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:52.113931    2561 projected.go:196] Error preparing data for projected volume kube-api-access-nmtr8 for pod openshift-network-diagnostics/network-check-target-v4ffn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:52.114064 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:52.113994    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8 podName:541880d8-884d-4644-9c35-3485805a5425 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:24.113975245 +0000 UTC m=+67.659265901 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmtr8" (UniqueName: "kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8") pod "network-check-target-v4ffn" (UID: "541880d8-884d-4644-9c35-3485805a5425") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 15:34:52.315921 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:52.315882    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:52.316077 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:52.315976    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:52.316077 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:52.316020    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 15:34:52.316077 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:52.316043    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbbbbbbb5-sqwnt: secret "image-registry-tls" not found
Sep 18 15:34:52.316207 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:52.316103    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls podName:b215b96a-f47b-4e8c-8df3-3c1aa336a57e nodeName:}" failed. No retries permitted until 2026-09-18 15:34:53.31608476 +0000 UTC m=+36.861375423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls") pod "image-registry-cbbbbbbb5-sqwnt" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e") : secret "image-registry-tls" not found
Sep 18 15:34:52.316207 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:52.316124    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 15:34:52.316207 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:52.316181    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls podName:ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:53.316164383 +0000 UTC m=+36.861455058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls") pod "dns-default-mmqzn" (UID: "ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407") : secret "dns-default-metrics-tls" not found
Sep 18 15:34:52.378317 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:52.377883    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:34:52.383649 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:52.383619    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 15:34:52.384116 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:52.384097    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nqjbs\""
Sep 18 15:34:52.385934 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:52.385911    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 15:34:53.323084 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:53.323051    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:53.323463 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:53.323114    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:53.323463 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:53.323201    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 15:34:53.323463 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:53.323212    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbbbbbbb5-sqwnt: secret "image-registry-tls" not found
Sep 18 15:34:53.323463 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:53.323201    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 15:34:53.323463 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:53.323258    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls podName:b215b96a-f47b-4e8c-8df3-3c1aa336a57e nodeName:}" failed. No retries permitted until 2026-09-18 15:34:55.323245613 +0000 UTC m=+38.868536265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls") pod "image-registry-cbbbbbbb5-sqwnt" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e") : secret "image-registry-tls" not found
Sep 18 15:34:53.323463 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:53.323315    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls podName:ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:55.32329917 +0000 UTC m=+38.868589835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls") pod "dns-default-mmqzn" (UID: "ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407") : secret "dns-default-metrics-tls" not found
Sep 18 15:34:53.373294 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:53.373260    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:34:53.373467 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:53.373448    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:34:53.377230 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:53.377213    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c572m\""
Sep 18 15:34:53.378101 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:53.378074    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 15:34:53.378209 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:53.378107    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 15:34:53.683678 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:53.683630    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rqkgz"]
Sep 18 15:34:53.692966 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:34:53.692942    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f7bd6b_71dd_48ea_b787_f0ed59ed4fa1.slice/crio-c5a240fb3c68f2ae654d938c03fc53dc0c2bdace1118381bf8fb7bc541ac5896 WatchSource:0}: Error finding container c5a240fb3c68f2ae654d938c03fc53dc0c2bdace1118381bf8fb7bc541ac5896: Status 404 returned error can't find the container with id c5a240fb3c68f2ae654d938c03fc53dc0c2bdace1118381bf8fb7bc541ac5896
Sep 18 15:34:54.547356 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:54.547322    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb4d3104-199c-402d-9f9e-ca63e562073c" containerID="f6f495078e2ef82b24b4960e18669d5c646ffc7b3e964ea944656abf35342e8f" exitCode=0
Sep 18 15:34:54.548397 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:54.547400    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zwstv" event={"ID":"bb4d3104-199c-402d-9f9e-ca63e562073c","Type":"ContainerDied","Data":"f6f495078e2ef82b24b4960e18669d5c646ffc7b3e964ea944656abf35342e8f"}
Sep 18 15:34:54.548969 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:54.548947    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rqkgz" event={"ID":"84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1","Type":"ContainerStarted","Data":"c5a240fb3c68f2ae654d938c03fc53dc0c2bdace1118381bf8fb7bc541ac5896"}
Sep 18 15:34:55.339363 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:55.339328    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:55.339529 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:55.339419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:55.339529 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:55.339496    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 15:34:55.339529 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:55.339518    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbbbbbbb5-sqwnt: secret "image-registry-tls" not found
Sep 18 15:34:55.339713 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:55.339532    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 15:34:55.339713 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:55.339581    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls podName:b215b96a-f47b-4e8c-8df3-3c1aa336a57e nodeName:}" failed. No retries permitted until 2026-09-18 15:34:59.339561456 +0000 UTC m=+42.884852135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls") pod "image-registry-cbbbbbbb5-sqwnt" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e") : secret "image-registry-tls" not found
Sep 18 15:34:55.339713 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:55.339597    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls podName:ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407 nodeName:}" failed. No retries permitted until 2026-09-18 15:34:59.339588912 +0000 UTC m=+42.884879565 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls") pod "dns-default-mmqzn" (UID: "ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407") : secret "dns-default-metrics-tls" not found
Sep 18 15:34:55.556594 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:55.556514    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb4d3104-199c-402d-9f9e-ca63e562073c" containerID="a65991cae1facc0615eb247b213158d263aed9a727b08af7b8979fa0eadba201" exitCode=0
Sep 18 15:34:55.556594 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:55.556566    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zwstv" event={"ID":"bb4d3104-199c-402d-9f9e-ca63e562073c","Type":"ContainerDied","Data":"a65991cae1facc0615eb247b213158d263aed9a727b08af7b8979fa0eadba201"}
Sep 18 15:34:56.562312 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:56.562082    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zwstv" event={"ID":"bb4d3104-199c-402d-9f9e-ca63e562073c","Type":"ContainerStarted","Data":"5f7408105b349a2c4b335e12a41ac75b29eb848accfbbd6e1b23425efcbb0a21"}
Sep 18 15:34:56.594890 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:56.594844    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zwstv" podStartSLOduration=8.98885801 podStartE2EDuration="38.594828829s" totalImagesPullingTime="29.605970819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:56.593485718 +0000 UTC m=+40.138776393" watchObservedRunningTime="2026-09-18 15:34:56.594828829 +0000 UTC m=+40.140119503"
Sep 18 15:34:57.566052 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:57.566011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rqkgz" event={"ID":"84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1","Type":"ContainerStarted","Data":"6db096db4a7ff6e03408429c78365f1d54ac38784975a08371b648d31b5e56e5"}
Sep 18 15:34:57.601173 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:57.601131    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rqkgz" podStartSLOduration=3.674185617 podStartE2EDuration="6.601117711s" totalImagesPullingTime="2.926932094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:34:57.600829014 +0000 UTC m=+41.146119687" watchObservedRunningTime="2026-09-18 15:34:57.601117711 +0000 UTC m=+41.146408385"
Sep 18 15:34:59.372042 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:59.372006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:34:59.372409 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:34:59.372066    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:34:59.372409 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:59.372144    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 15:34:59.372409 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:59.372163    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbbbbbbb5-sqwnt: secret "image-registry-tls" not found
Sep 18 15:34:59.372409 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:59.372149    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 15:34:59.372409 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:59.372212    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls podName:b215b96a-f47b-4e8c-8df3-3c1aa336a57e nodeName:}" failed. No retries permitted until 2026-09-18 15:35:07.372197344 +0000 UTC m=+50.917487995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls") pod "image-registry-cbbbbbbb5-sqwnt" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e") : secret "image-registry-tls" not found
Sep 18 15:34:59.372409 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:34:59.372225    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls podName:ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:07.372219299 +0000 UTC m=+50.917509951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls") pod "dns-default-mmqzn" (UID: "ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407") : secret "dns-default-metrics-tls" not found
Sep 18 15:35:07.429515 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.429472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:35:07.429941 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.429549    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:35:07.429941 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:07.429626    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 15:35:07.429941 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:07.429698    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 15:35:07.429941 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:07.429709    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls podName:ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:23.429693759 +0000 UTC m=+66.974984410 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls") pod "dns-default-mmqzn" (UID: "ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407") : secret "dns-default-metrics-tls" not found
Sep 18 15:35:07.429941 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:07.429713    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbbbbbbb5-sqwnt: secret "image-registry-tls" not found
Sep 18 15:35:07.429941 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:07.429764    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls podName:b215b96a-f47b-4e8c-8df3-3c1aa336a57e nodeName:}" failed. No retries permitted until 2026-09-18 15:35:23.42974789 +0000 UTC m=+66.975038557 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls") pod "image-registry-cbbbbbbb5-sqwnt" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e") : secret "image-registry-tls" not found
Sep 18 15:35:07.495119 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.495098    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-k4r5l"]
Sep 18 15:35:07.498221 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.498207    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-k4r5l"
Sep 18 15:35:07.503093 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.503074    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 15:35:07.503215 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.503170    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tzmp9\""
Sep 18 15:35:07.503284 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.503266    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 15:35:07.511550 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.511532    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"]
Sep 18 15:35:07.514791 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.514777    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"
Sep 18 15:35:07.518163 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.518141    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-2965g"]
Sep 18 15:35:07.521230 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.521211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.523737 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.523719    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 18 15:35:07.523971 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.523955    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 18 15:35:07.524015 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.523962    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 18 15:35:07.525112 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.525098    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 18 15:35:07.525321 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.525309    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vx52s\""
Sep 18 15:35:07.525435 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.525422    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 15:35:07.526190 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.526167    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4gtxv\""
Sep 18 15:35:07.526268 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.526174    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 18 15:35:07.526456 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.526440    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 15:35:07.537485 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.537466    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 18 15:35:07.538928 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.538913    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-k4r5l"]
Sep 18 15:35:07.539479 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.539463    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"]
Sep 18 15:35:07.557520 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.557498    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-2965g"]
Sep 18 15:35:07.603473 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.603449    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"]
Sep 18 15:35:07.606657 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.606644    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:07.609796 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.609771    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 15:35:07.612576 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.612555    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 15:35:07.612833 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.612816    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-psf8j\""
Sep 18 15:35:07.612935 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.612829    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 18 15:35:07.612935 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.612830    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 18 15:35:07.628820 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.628798    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"]
Sep 18 15:35:07.630215 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.630193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f4c511-6c8c-45f6-a86d-bb6f5d204022-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.630297 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.630234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79qtc\" (UniqueName: \"kubernetes.io/projected/d0d4b309-4771-4000-8bdd-70ee7489136b-kube-api-access-79qtc\") pod \"cluster-samples-operator-685d7d68b5-77rbk\" (UID: \"d0d4b309-4771-4000-8bdd-70ee7489136b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"
Sep 18 15:35:07.630335 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.630294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42f4c511-6c8c-45f6-a86d-bb6f5d204022-serving-cert\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.630335 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.630317    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfcrh\" (UniqueName: \"kubernetes.io/projected/42f4c511-6c8c-45f6-a86d-bb6f5d204022-kube-api-access-nfcrh\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.630399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.630346    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/42f4c511-6c8c-45f6-a86d-bb6f5d204022-snapshots\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.630399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.630371    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f4c511-6c8c-45f6-a86d-bb6f5d204022-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.630466 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.630453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77rbk\" (UID: \"d0d4b309-4771-4000-8bdd-70ee7489136b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"
Sep 18 15:35:07.630508 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.630478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f4c511-6c8c-45f6-a86d-bb6f5d204022-tmp\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.630571 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.630554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fvmd\" (UniqueName: \"kubernetes.io/projected/67e40f73-83c9-4f84-81d8-3b11ef6814d5-kube-api-access-8fvmd\") pod \"volume-data-source-validator-5c4db8bdf9-k4r5l\" (UID: \"67e40f73-83c9-4f84-81d8-3b11ef6814d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-k4r5l"
Sep 18 15:35:07.701680 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.701604    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"]
Sep 18 15:35:07.704899 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.704886    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"
Sep 18 15:35:07.708063 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.708041    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 15:35:07.708167 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.708098    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-r5hgh\""
Sep 18 15:35:07.708240 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.708222    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 15:35:07.708464 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.708445    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 15:35:07.708655 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.708641    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 15:35:07.715618 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.715558    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-7qzjj"]
Sep 18 15:35:07.718567 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.718550    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-db9cx"]
Sep 18 15:35:07.718709 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.718656    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7qzjj"
Sep 18 15:35:07.721471 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.721450    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:07.723177 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.723158    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2zq2t\""
Sep 18 15:35:07.723409 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.723392    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"]
Sep 18 15:35:07.724397 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.724379    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-mzxxg\""
Sep 18 15:35:07.724397 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.724395    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 18 15:35:07.724892 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.724874    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 18 15:35:07.725530 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.725515    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 18 15:35:07.725624 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.725572    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 18 15:35:07.726724 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.726707    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6bf6bd4c64-ppwrg"]
Sep 18 15:35:07.726820 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.726801    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"
Sep 18 15:35:07.729484 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.729471    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"]
Sep 18 15:35:07.729546 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.729531    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.730913 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.730898    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f4c511-6c8c-45f6-a86d-bb6f5d204022-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.730999 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.730926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb4cm\" (UniqueName: \"kubernetes.io/projected/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-kube-api-access-xb4cm\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:07.730999 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.730947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79qtc\" (UniqueName: \"kubernetes.io/projected/d0d4b309-4771-4000-8bdd-70ee7489136b-kube-api-access-79qtc\") pod \"cluster-samples-operator-685d7d68b5-77rbk\" (UID: \"d0d4b309-4771-4000-8bdd-70ee7489136b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"
Sep 18 15:35:07.730999 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.730972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:07.731163 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.731100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42f4c511-6c8c-45f6-a86d-bb6f5d204022-serving-cert\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.731163 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.731126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfcrh\" (UniqueName: \"kubernetes.io/projected/42f4c511-6c8c-45f6-a86d-bb6f5d204022-kube-api-access-nfcrh\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.731163 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.731146    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/42f4c511-6c8c-45f6-a86d-bb6f5d204022-snapshots\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.731163 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.731161    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f4c511-6c8c-45f6-a86d-bb6f5d204022-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.731367 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.731271    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77rbk\" (UID: \"d0d4b309-4771-4000-8bdd-70ee7489136b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"
Sep 18 15:35:07.731367 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.731295    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f4c511-6c8c-45f6-a86d-bb6f5d204022-tmp\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.731367 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.731321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:07.731521 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.731386    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fvmd\" (UniqueName: \"kubernetes.io/projected/67e40f73-83c9-4f84-81d8-3b11ef6814d5-kube-api-access-8fvmd\") pod \"volume-data-source-validator-5c4db8bdf9-k4r5l\" (UID: \"67e40f73-83c9-4f84-81d8-3b11ef6814d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-k4r5l"
Sep 18 15:35:07.731603 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:07.731586    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 15:35:07.731706 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:07.731687    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls podName:d0d4b309-4771-4000-8bdd-70ee7489136b nodeName:}" failed. No retries permitted until 2026-09-18 15:35:08.231650593 +0000 UTC m=+51.776941264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-77rbk" (UID: "d0d4b309-4771-4000-8bdd-70ee7489136b") : secret "samples-operator-tls" not found
Sep 18 15:35:07.731840 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.731821    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/42f4c511-6c8c-45f6-a86d-bb6f5d204022-snapshots\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.731995 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.731978    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f4c511-6c8c-45f6-a86d-bb6f5d204022-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.732176 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.732051    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42f4c511-6c8c-45f6-a86d-bb6f5d204022-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.732452 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.732409    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42f4c511-6c8c-45f6-a86d-bb6f5d204022-tmp\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.734636 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.734616    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/42f4c511-6c8c-45f6-a86d-bb6f5d204022-serving-cert\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.734784 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.734767    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 18 15:35:07.734841 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.734780    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 18 15:35:07.734894 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.734879    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 18 15:35:07.734987 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.734968    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 18 15:35:07.735212 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.735197    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 15:35:07.735265 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.735229    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 15:35:07.735265 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.735248    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 15:35:07.735359 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.735273    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 15:35:07.735416 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.735407    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9gvdh\""
Sep 18 15:35:07.735463 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.735419    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 15:35:07.735875 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.735858    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jwwgn\""
Sep 18 15:35:07.736302 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.736268    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-7qzjj"]
Sep 18 15:35:07.736601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.736585    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 18 15:35:07.739404 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.739385    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-db9cx"]
Sep 18 15:35:07.740700 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.740683    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 15:35:07.744531 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.744508    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"]
Sep 18 15:35:07.754817 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.754795    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfcrh\" (UniqueName: \"kubernetes.io/projected/42f4c511-6c8c-45f6-a86d-bb6f5d204022-kube-api-access-nfcrh\") pod \"insights-operator-6f9c644dd9-2965g\" (UID: \"42f4c511-6c8c-45f6-a86d-bb6f5d204022\") " pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.755404 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.755388    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fvmd\" (UniqueName: \"kubernetes.io/projected/67e40f73-83c9-4f84-81d8-3b11ef6814d5-kube-api-access-8fvmd\") pod \"volume-data-source-validator-5c4db8bdf9-k4r5l\" (UID: \"67e40f73-83c9-4f84-81d8-3b11ef6814d5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-k4r5l"
Sep 18 15:35:07.766092 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.766073    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6bf6bd4c64-ppwrg"]
Sep 18 15:35:07.769137 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.769093    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79qtc\" (UniqueName: \"kubernetes.io/projected/d0d4b309-4771-4000-8bdd-70ee7489136b-kube-api-access-79qtc\") pod \"cluster-samples-operator-685d7d68b5-77rbk\" (UID: \"d0d4b309-4771-4000-8bdd-70ee7489136b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"
Sep 18 15:35:07.806605 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.806582    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-k4r5l"
Sep 18 15:35:07.829085 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.829067    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-2965g"
Sep 18 15:35:07.831894 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.831873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.831994 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.831908    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmd7c\" (UniqueName: \"kubernetes.io/projected/0eea31a1-252d-4d6a-a061-6865d90ad1e0-kube-api-access-dmd7c\") pod \"network-check-source-7cbc45fddc-7qzjj\" (UID: \"0eea31a1-252d-4d6a-a061-6865d90ad1e0\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7qzjj"
Sep 18 15:35:07.831994 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.831970    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:07.832078 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/577230bb-9bb3-42c4-9e6f-dda006b1552f-trusted-ca\") pod \"console-operator-fd4bb46c-db9cx\" (UID: \"577230bb-9bb3-42c4-9e6f-dda006b1552f\") " pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:07.832078 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832059    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.832277 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832099    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2def30a4-9a64-4048-8d1f-90b6ae7f3075-config\") pod \"service-ca-operator-57cdc4b768-vfzz9\" (UID: \"2def30a4-9a64-4048-8d1f-90b6ae7f3075\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"
Sep 18 15:35:07.832277 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-stats-auth\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.832277 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32803062-157c-4605-8474-a32715b72352-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-xn2fc\" (UID: \"32803062-157c-4605-8474-a32715b72352\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"
Sep 18 15:35:07.832277 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f56mj\" (UniqueName: \"kubernetes.io/projected/32803062-157c-4605-8474-a32715b72352-kube-api-access-f56mj\") pod \"kube-storage-version-migrator-operator-55889f7b47-xn2fc\" (UID: \"32803062-157c-4605-8474-a32715b72352\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"
Sep 18 15:35:07.832277 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832264    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:07.832528 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832292    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/577230bb-9bb3-42c4-9e6f-dda006b1552f-serving-cert\") pod \"console-operator-fd4bb46c-db9cx\" (UID: \"577230bb-9bb3-42c4-9e6f-dda006b1552f\") " pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:07.832528 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2def30a4-9a64-4048-8d1f-90b6ae7f3075-serving-cert\") pod \"service-ca-operator-57cdc4b768-vfzz9\" (UID: \"2def30a4-9a64-4048-8d1f-90b6ae7f3075\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"
Sep 18 15:35:07.832528 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbktj\" (UniqueName: \"kubernetes.io/projected/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-kube-api-access-mbktj\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.832528 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:07.832423    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 15:35:07.832528 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2mzq\" (UniqueName: \"kubernetes.io/projected/2def30a4-9a64-4048-8d1f-90b6ae7f3075-kube-api-access-m2mzq\") pod \"service-ca-operator-57cdc4b768-vfzz9\" (UID: \"2def30a4-9a64-4048-8d1f-90b6ae7f3075\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"
Sep 18 15:35:07.832528 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:07.832490    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls podName:52dadbd4-3e28-42d7-9d38-b4343c8c37ab nodeName:}" failed. No retries permitted until 2026-09-18 15:35:08.332468009 +0000 UTC m=+51.877758678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-czd8s" (UID: "52dadbd4-3e28-42d7-9d38-b4343c8c37ab") : secret "cluster-monitoring-operator-tls" not found
Sep 18 15:35:07.832528 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32803062-157c-4605-8474-a32715b72352-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-xn2fc\" (UID: \"32803062-157c-4605-8474-a32715b72352\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"
Sep 18 15:35:07.832915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832544    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/577230bb-9bb3-42c4-9e6f-dda006b1552f-config\") pod \"console-operator-fd4bb46c-db9cx\" (UID: \"577230bb-9bb3-42c4-9e6f-dda006b1552f\") " pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:07.832915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-default-certificate\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.832915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832763    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb4cm\" (UniqueName: \"kubernetes.io/projected/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-kube-api-access-xb4cm\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:07.832915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stnvr\" (UniqueName: \"kubernetes.io/projected/577230bb-9bb3-42c4-9e6f-dda006b1552f-kube-api-access-stnvr\") pod \"console-operator-fd4bb46c-db9cx\" (UID: \"577230bb-9bb3-42c4-9e6f-dda006b1552f\") " pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:07.832915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.832888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:07.842993 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.842967    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb4cm\" (UniqueName: \"kubernetes.io/projected/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-kube-api-access-xb4cm\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:07.933314 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnvr\" (UniqueName: \"kubernetes.io/projected/577230bb-9bb3-42c4-9e6f-dda006b1552f-kube-api-access-stnvr\") pod \"console-operator-fd4bb46c-db9cx\" (UID: \"577230bb-9bb3-42c4-9e6f-dda006b1552f\") " pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:07.933454 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933324    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.933454 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmd7c\" (UniqueName: \"kubernetes.io/projected/0eea31a1-252d-4d6a-a061-6865d90ad1e0-kube-api-access-dmd7c\") pod \"network-check-source-7cbc45fddc-7qzjj\" (UID: \"0eea31a1-252d-4d6a-a061-6865d90ad1e0\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7qzjj"
Sep 18 15:35:07.933581 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:07.933460    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 15:35:07.933581 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:07.933537    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs podName:3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:08.433515619 +0000 UTC m=+51.978806289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs") pod "router-default-6bf6bd4c64-ppwrg" (UID: "3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436") : secret "router-metrics-certs-default" not found
Sep 18 15:35:07.933727 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933579    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/577230bb-9bb3-42c4-9e6f-dda006b1552f-trusted-ca\") pod \"console-operator-fd4bb46c-db9cx\" (UID: \"577230bb-9bb3-42c4-9e6f-dda006b1552f\") " pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:07.933727 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933616    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.933727 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2def30a4-9a64-4048-8d1f-90b6ae7f3075-config\") pod \"service-ca-operator-57cdc4b768-vfzz9\" (UID: \"2def30a4-9a64-4048-8d1f-90b6ae7f3075\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"
Sep 18 15:35:07.933727 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933706    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-stats-auth\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.933910 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:07.933778    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle podName:3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:08.433765319 +0000 UTC m=+51.979055971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle") pod "router-default-6bf6bd4c64-ppwrg" (UID: "3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436") : configmap references non-existent config key: service-ca.crt
Sep 18 15:35:07.933910 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933812    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32803062-157c-4605-8474-a32715b72352-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-xn2fc\" (UID: \"32803062-157c-4605-8474-a32715b72352\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"
Sep 18 15:35:07.933910 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f56mj\" (UniqueName: \"kubernetes.io/projected/32803062-157c-4605-8474-a32715b72352-kube-api-access-f56mj\") pod \"kube-storage-version-migrator-operator-55889f7b47-xn2fc\" (UID: \"32803062-157c-4605-8474-a32715b72352\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"
Sep 18 15:35:07.933910 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/577230bb-9bb3-42c4-9e6f-dda006b1552f-serving-cert\") pod \"console-operator-fd4bb46c-db9cx\" (UID: \"577230bb-9bb3-42c4-9e6f-dda006b1552f\") " pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:07.934131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2def30a4-9a64-4048-8d1f-90b6ae7f3075-serving-cert\") pod \"service-ca-operator-57cdc4b768-vfzz9\" (UID: \"2def30a4-9a64-4048-8d1f-90b6ae7f3075\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"
Sep 18 15:35:07.934131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbktj\" (UniqueName: \"kubernetes.io/projected/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-kube-api-access-mbktj\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.934131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933973    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2mzq\" (UniqueName: \"kubernetes.io/projected/2def30a4-9a64-4048-8d1f-90b6ae7f3075-kube-api-access-m2mzq\") pod \"service-ca-operator-57cdc4b768-vfzz9\" (UID: \"2def30a4-9a64-4048-8d1f-90b6ae7f3075\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"
Sep 18 15:35:07.934131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.933996    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32803062-157c-4605-8474-a32715b72352-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-xn2fc\" (UID: \"32803062-157c-4605-8474-a32715b72352\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"
Sep 18 15:35:07.934131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.934026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/577230bb-9bb3-42c4-9e6f-dda006b1552f-config\") pod \"console-operator-fd4bb46c-db9cx\" (UID: \"577230bb-9bb3-42c4-9e6f-dda006b1552f\") " pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:07.934131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.934058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-default-certificate\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.934614 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.934566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/577230bb-9bb3-42c4-9e6f-dda006b1552f-trusted-ca\") pod \"console-operator-fd4bb46c-db9cx\" (UID: \"577230bb-9bb3-42c4-9e6f-dda006b1552f\") " pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:07.934732 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.934689    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/32803062-157c-4605-8474-a32715b72352-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-xn2fc\" (UID: \"32803062-157c-4605-8474-a32715b72352\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"
Sep 18 15:35:07.934958 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.934939    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2def30a4-9a64-4048-8d1f-90b6ae7f3075-config\") pod \"service-ca-operator-57cdc4b768-vfzz9\" (UID: \"2def30a4-9a64-4048-8d1f-90b6ae7f3075\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"
Sep 18 15:35:07.935142 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.935114    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/577230bb-9bb3-42c4-9e6f-dda006b1552f-config\") pod \"console-operator-fd4bb46c-db9cx\" (UID: \"577230bb-9bb3-42c4-9e6f-dda006b1552f\") " pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:07.936561 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.936538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/32803062-157c-4605-8474-a32715b72352-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-xn2fc\" (UID: \"32803062-157c-4605-8474-a32715b72352\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"
Sep 18 15:35:07.936647 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.936610    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2def30a4-9a64-4048-8d1f-90b6ae7f3075-serving-cert\") pod \"service-ca-operator-57cdc4b768-vfzz9\" (UID: \"2def30a4-9a64-4048-8d1f-90b6ae7f3075\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"
Sep 18 15:35:07.936647 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.936621    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/577230bb-9bb3-42c4-9e6f-dda006b1552f-serving-cert\") pod \"console-operator-fd4bb46c-db9cx\" (UID: \"577230bb-9bb3-42c4-9e6f-dda006b1552f\") " pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:07.936851 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.936829    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-stats-auth\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.936851 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.936844    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-default-certificate\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.949623 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.949603    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnvr\" (UniqueName: \"kubernetes.io/projected/577230bb-9bb3-42c4-9e6f-dda006b1552f-kube-api-access-stnvr\") pod \"console-operator-fd4bb46c-db9cx\" (UID: \"577230bb-9bb3-42c4-9e6f-dda006b1552f\") " pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:07.950901 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.950884    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f56mj\" (UniqueName: \"kubernetes.io/projected/32803062-157c-4605-8474-a32715b72352-kube-api-access-f56mj\") pod \"kube-storage-version-migrator-operator-55889f7b47-xn2fc\" (UID: \"32803062-157c-4605-8474-a32715b72352\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"
Sep 18 15:35:07.952915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.952867    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmd7c\" (UniqueName: \"kubernetes.io/projected/0eea31a1-252d-4d6a-a061-6865d90ad1e0-kube-api-access-dmd7c\") pod \"network-check-source-7cbc45fddc-7qzjj\" (UID: \"0eea31a1-252d-4d6a-a061-6865d90ad1e0\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7qzjj"
Sep 18 15:35:07.953241 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.953220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbktj\" (UniqueName: \"kubernetes.io/projected/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-kube-api-access-mbktj\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:07.954623 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:07.954602    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2mzq\" (UniqueName: \"kubernetes.io/projected/2def30a4-9a64-4048-8d1f-90b6ae7f3075-kube-api-access-m2mzq\") pod \"service-ca-operator-57cdc4b768-vfzz9\" (UID: \"2def30a4-9a64-4048-8d1f-90b6ae7f3075\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"
Sep 18 15:35:08.013760 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.013742    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"
Sep 18 15:35:08.029234 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.029213    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7qzjj"
Sep 18 15:35:08.035308 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.035288    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:08.048171 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.048147    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"
Sep 18 15:35:08.112379 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.112177    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-k4r5l"]
Sep 18 15:35:08.117107 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:08.116753    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e40f73_83c9_4f84_81d8_3b11ef6814d5.slice/crio-b43f691365ac9c6154a8c122c83900ed722f0c50f6547f83e77497250a4cce7e WatchSource:0}: Error finding container b43f691365ac9c6154a8c122c83900ed722f0c50f6547f83e77497250a4cce7e: Status 404 returned error can't find the container with id b43f691365ac9c6154a8c122c83900ed722f0c50f6547f83e77497250a4cce7e
Sep 18 15:35:08.117611 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.117535    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-2965g"]
Sep 18 15:35:08.236562 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.236467    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77rbk\" (UID: \"d0d4b309-4771-4000-8bdd-70ee7489136b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"
Sep 18 15:35:08.236715 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:08.236609    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 15:35:08.236715 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:08.236691    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls podName:d0d4b309-4771-4000-8bdd-70ee7489136b nodeName:}" failed. No retries permitted until 2026-09-18 15:35:09.236657902 +0000 UTC m=+52.781948585 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-77rbk" (UID: "d0d4b309-4771-4000-8bdd-70ee7489136b") : secret "samples-operator-tls" not found
Sep 18 15:35:08.297464 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.297436    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc"]
Sep 18 15:35:08.300950 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:08.300907    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32803062_157c_4605_8474_a32715b72352.slice/crio-5513f1bf9659120bffbac334f0e4b50738572fd077be0d7b8a736626718807e4 WatchSource:0}: Error finding container 5513f1bf9659120bffbac334f0e4b50738572fd077be0d7b8a736626718807e4: Status 404 returned error can't find the container with id 5513f1bf9659120bffbac334f0e4b50738572fd077be0d7b8a736626718807e4
Sep 18 15:35:08.337797 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.337771    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:08.337919 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:08.337904    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 15:35:08.337967 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:08.337958    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls podName:52dadbd4-3e28-42d7-9d38-b4343c8c37ab nodeName:}" failed. No retries permitted until 2026-09-18 15:35:09.337942403 +0000 UTC m=+52.883233055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-czd8s" (UID: "52dadbd4-3e28-42d7-9d38-b4343c8c37ab") : secret "cluster-monitoring-operator-tls" not found
Sep 18 15:35:08.347352 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.347334    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-7qzjj"]
Sep 18 15:35:08.348582 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.348558    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9"]
Sep 18 15:35:08.349530 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:08.349501    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eea31a1_252d_4d6a_a061_6865d90ad1e0.slice/crio-962145bfc602460fd64571dfd2ba9e85adebe4116d55276fd265fa7a32263f93 WatchSource:0}: Error finding container 962145bfc602460fd64571dfd2ba9e85adebe4116d55276fd265fa7a32263f93: Status 404 returned error can't find the container with id 962145bfc602460fd64571dfd2ba9e85adebe4116d55276fd265fa7a32263f93
Sep 18 15:35:08.349770 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.349750    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-db9cx"]
Sep 18 15:35:08.350426 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:08.350396    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2def30a4_9a64_4048_8d1f_90b6ae7f3075.slice/crio-a2ced3905a7b43eb31b4641f074c838eccb46d93732c1ce9894bb2771e116c55 WatchSource:0}: Error finding container a2ced3905a7b43eb31b4641f074c838eccb46d93732c1ce9894bb2771e116c55: Status 404 returned error can't find the container with id a2ced3905a7b43eb31b4641f074c838eccb46d93732c1ce9894bb2771e116c55
Sep 18 15:35:08.353405 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:08.353389    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577230bb_9bb3_42c4_9e6f_dda006b1552f.slice/crio-aaa23380342de3298012dd4910b9d07a790a255356000040ae454afe2ed424df WatchSource:0}: Error finding container aaa23380342de3298012dd4910b9d07a790a255356000040ae454afe2ed424df: Status 404 returned error can't find the container with id aaa23380342de3298012dd4910b9d07a790a255356000040ae454afe2ed424df
Sep 18 15:35:08.438788 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.438768    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:08.439070 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.438820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:08.439070 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:08.438903    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 15:35:08.439070 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:08.438946    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle podName:3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:09.438934224 +0000 UTC m=+52.984224889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle") pod "router-default-6bf6bd4c64-ppwrg" (UID: "3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436") : configmap references non-existent config key: service-ca.crt
Sep 18 15:35:08.439070 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:08.438959    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs podName:3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:09.438953781 +0000 UTC m=+52.984244432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs") pod "router-default-6bf6bd4c64-ppwrg" (UID: "3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436") : secret "router-metrics-certs-default" not found
Sep 18 15:35:08.591178 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.591108    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9" event={"ID":"2def30a4-9a64-4048-8d1f-90b6ae7f3075","Type":"ContainerStarted","Data":"a2ced3905a7b43eb31b4641f074c838eccb46d93732c1ce9894bb2771e116c55"}
Sep 18 15:35:08.592225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.592203    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx" event={"ID":"577230bb-9bb3-42c4-9e6f-dda006b1552f","Type":"ContainerStarted","Data":"aaa23380342de3298012dd4910b9d07a790a255356000040ae454afe2ed424df"}
Sep 18 15:35:08.593228 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.593207    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7qzjj" event={"ID":"0eea31a1-252d-4d6a-a061-6865d90ad1e0","Type":"ContainerStarted","Data":"962145bfc602460fd64571dfd2ba9e85adebe4116d55276fd265fa7a32263f93"}
Sep 18 15:35:08.594247 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.594228    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc" event={"ID":"32803062-157c-4605-8474-a32715b72352","Type":"ContainerStarted","Data":"5513f1bf9659120bffbac334f0e4b50738572fd077be0d7b8a736626718807e4"}
Sep 18 15:35:08.595304 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.595286    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-2965g" event={"ID":"42f4c511-6c8c-45f6-a86d-bb6f5d204022","Type":"ContainerStarted","Data":"d6015d21d87448158d88b29eb0112ca7d19eb9af04345a7b0d13e06a2e762134"}
Sep 18 15:35:08.596271 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:08.596251    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-k4r5l" event={"ID":"67e40f73-83c9-4f84-81d8-3b11ef6814d5","Type":"ContainerStarted","Data":"b43f691365ac9c6154a8c122c83900ed722f0c50f6547f83e77497250a4cce7e"}
Sep 18 15:35:09.247268 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:09.247178    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77rbk\" (UID: \"d0d4b309-4771-4000-8bdd-70ee7489136b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"
Sep 18 15:35:09.247467 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:09.247301    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 15:35:09.247467 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:09.247368    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls podName:d0d4b309-4771-4000-8bdd-70ee7489136b nodeName:}" failed. No retries permitted until 2026-09-18 15:35:11.247350893 +0000 UTC m=+54.792641569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-77rbk" (UID: "d0d4b309-4771-4000-8bdd-70ee7489136b") : secret "samples-operator-tls" not found
Sep 18 15:35:09.347946 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:09.347908    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:09.348194 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:09.348082    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 15:35:09.348194 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:09.348149    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls podName:52dadbd4-3e28-42d7-9d38-b4343c8c37ab nodeName:}" failed. No retries permitted until 2026-09-18 15:35:11.348128411 +0000 UTC m=+54.893419081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-czd8s" (UID: "52dadbd4-3e28-42d7-9d38-b4343c8c37ab") : secret "cluster-monitoring-operator-tls" not found
Sep 18 15:35:09.449197 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:09.448345    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:09.449197 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:09.448442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:09.449197 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:09.448695    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle podName:3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:11.448673949 +0000 UTC m=+54.993964619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle") pod "router-default-6bf6bd4c64-ppwrg" (UID: "3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436") : configmap references non-existent config key: service-ca.crt
Sep 18 15:35:09.449197 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:09.449107    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 15:35:09.449197 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:09.449162    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs podName:3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:11.449143596 +0000 UTC m=+54.994434255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs") pod "router-default-6bf6bd4c64-ppwrg" (UID: "3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436") : secret "router-metrics-certs-default" not found
Sep 18 15:35:11.268790 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:11.268713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77rbk\" (UID: \"d0d4b309-4771-4000-8bdd-70ee7489136b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"
Sep 18 15:35:11.269184 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:11.268896    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 15:35:11.269184 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:11.268969    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls podName:d0d4b309-4771-4000-8bdd-70ee7489136b nodeName:}" failed. No retries permitted until 2026-09-18 15:35:15.268952915 +0000 UTC m=+58.814243567 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-77rbk" (UID: "d0d4b309-4771-4000-8bdd-70ee7489136b") : secret "samples-operator-tls" not found
Sep 18 15:35:11.370015 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:11.369980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:11.370201 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:11.370102    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 15:35:11.370266 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:11.370230    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls podName:52dadbd4-3e28-42d7-9d38-b4343c8c37ab nodeName:}" failed. No retries permitted until 2026-09-18 15:35:15.370204931 +0000 UTC m=+58.915495595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-czd8s" (UID: "52dadbd4-3e28-42d7-9d38-b4343c8c37ab") : secret "cluster-monitoring-operator-tls" not found
Sep 18 15:35:11.470755 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:11.470720    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:11.470909 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:11.470820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:11.470909 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:11.470903    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 15:35:11.471030 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:11.470911    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle podName:3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:15.470887779 +0000 UTC m=+59.016178454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle") pod "router-default-6bf6bd4c64-ppwrg" (UID: "3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436") : configmap references non-existent config key: service-ca.crt
Sep 18 15:35:11.471030 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:11.470940    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs podName:3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:15.47093 +0000 UTC m=+59.016220652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs") pod "router-default-6bf6bd4c64-ppwrg" (UID: "3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436") : secret "router-metrics-certs-default" not found
Sep 18 15:35:15.304798 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.304759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77rbk\" (UID: \"d0d4b309-4771-4000-8bdd-70ee7489136b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"
Sep 18 15:35:15.305283 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:15.304934    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 18 15:35:15.305283 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:15.305014    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls podName:d0d4b309-4771-4000-8bdd-70ee7489136b nodeName:}" failed. No retries permitted until 2026-09-18 15:35:23.304992782 +0000 UTC m=+66.850283437 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls") pod "cluster-samples-operator-685d7d68b5-77rbk" (UID: "d0d4b309-4771-4000-8bdd-70ee7489136b") : secret "samples-operator-tls" not found
Sep 18 15:35:15.406128 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.406083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:15.406300 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:15.406261    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 15:35:15.406373 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:15.406342    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls podName:52dadbd4-3e28-42d7-9d38-b4343c8c37ab nodeName:}" failed. No retries permitted until 2026-09-18 15:35:23.406319579 +0000 UTC m=+66.951610252 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-czd8s" (UID: "52dadbd4-3e28-42d7-9d38-b4343c8c37ab") : secret "cluster-monitoring-operator-tls" not found
Sep 18 15:35:15.506834 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.506802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:15.507004 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.506872    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:15.507004 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:15.506987    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 15:35:15.507105 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:15.507045    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle podName:3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:23.507027691 +0000 UTC m=+67.052318360 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle") pod "router-default-6bf6bd4c64-ppwrg" (UID: "3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436") : configmap references non-existent config key: service-ca.crt
Sep 18 15:35:15.507170 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:15.507125    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs podName:3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:23.507104183 +0000 UTC m=+67.052394836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs") pod "router-default-6bf6bd4c64-ppwrg" (UID: "3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436") : secret "router-metrics-certs-default" not found
Sep 18 15:35:15.624110 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.624079    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9" event={"ID":"2def30a4-9a64-4048-8d1f-90b6ae7f3075","Type":"ContainerStarted","Data":"3dee9ec1bd8dc3c7eed8991bcb49029d990d75ad081aa5cc343a95316384ce78"}
Sep 18 15:35:15.625991 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.625967    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/0.log"
Sep 18 15:35:15.626090 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.626009    2561 generic.go:356] "Generic (PLEG): container finished" podID="577230bb-9bb3-42c4-9e6f-dda006b1552f" containerID="81f475253c0f6c9ce47ce3607ada5d777f1b8a8a5d3ce74d3790204c21cb680f" exitCode=255
Sep 18 15:35:15.626090 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.626077    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx" event={"ID":"577230bb-9bb3-42c4-9e6f-dda006b1552f","Type":"ContainerDied","Data":"81f475253c0f6c9ce47ce3607ada5d777f1b8a8a5d3ce74d3790204c21cb680f"}
Sep 18 15:35:15.626289 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.626267    2561 scope.go:117] "RemoveContainer" containerID="81f475253c0f6c9ce47ce3607ada5d777f1b8a8a5d3ce74d3790204c21cb680f"
Sep 18 15:35:15.627883 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.627862    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7qzjj" event={"ID":"0eea31a1-252d-4d6a-a061-6865d90ad1e0","Type":"ContainerStarted","Data":"7cfd2cef1e659b847056b0e6bd503a8558255df939ba3b56193b69c632b1e4b9"}
Sep 18 15:35:15.629620 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.629596    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc" event={"ID":"32803062-157c-4605-8474-a32715b72352","Type":"ContainerStarted","Data":"9768dc93c04f9dc48a5d19aadb94442b717f3062cc17161e434299c0aaec2f86"}
Sep 18 15:35:15.631232 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.631210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-2965g" event={"ID":"42f4c511-6c8c-45f6-a86d-bb6f5d204022","Type":"ContainerStarted","Data":"ad6a6723fbf2e0ba5e19657af9e727159b3e886cb19973b0f2d2a4a4e48a72ac"}
Sep 18 15:35:15.632727 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.632710    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-k4r5l" event={"ID":"67e40f73-83c9-4f84-81d8-3b11ef6814d5","Type":"ContainerStarted","Data":"96305c589b0c93967457427fdb3960bedac13b91b86e9d05f6c52dbdb65a231e"}
Sep 18 15:35:15.679360 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.679246    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9" podStartSLOduration=2.498381899 podStartE2EDuration="8.679230808s" totalImagesPullingTime="6.180848909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:15.649471025 +0000 UTC m=+59.194761701" watchObservedRunningTime="2026-09-18 15:35:15.679230808 +0000 UTC m=+59.224521483"
Sep 18 15:35:15.679822 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.679787    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-7qzjj" podStartSLOduration=2.48876684 podStartE2EDuration="8.679772749s" totalImagesPullingTime="6.191005909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:15.679641114 +0000 UTC m=+59.224931789" watchObservedRunningTime="2026-09-18 15:35:15.679772749 +0000 UTC m=+59.225063420"
Sep 18 15:35:15.697093 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.697051    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-k4r5l" podStartSLOduration=2.349250777 podStartE2EDuration="8.697021117s" totalImagesPullingTime="6.34777034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:15.696962546 +0000 UTC m=+59.242253220" watchObservedRunningTime="2026-09-18 15:35:15.697021117 +0000 UTC m=+59.242311791"
Sep 18 15:35:15.722222 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.722172    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-2965g" podStartSLOduration=2.312105707 podStartE2EDuration="8.722155069s" totalImagesPullingTime="6.410049362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:15.72068807 +0000 UTC m=+59.265978744" watchObservedRunningTime="2026-09-18 15:35:15.722155069 +0000 UTC m=+59.267445746"
Sep 18 15:35:15.745112 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.745066    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc" podStartSLOduration=2.513372209 podStartE2EDuration="8.745047436s" totalImagesPullingTime="6.231675227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:15.744565972 +0000 UTC m=+59.289856646" watchObservedRunningTime="2026-09-18 15:35:15.745047436 +0000 UTC m=+59.290338111"
Sep 18 15:35:15.989772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:15.989745    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8rgc2"]
Sep 18 15:35:16.013237 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.013208    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8rgc2"]
Sep 18 15:35:16.013389 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.013377    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.016177 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.016148    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 15:35:16.016295 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.016213    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c72vd\""
Sep 18 15:35:16.016733 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.016706    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 15:35:16.113730 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.113696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4192f75-9684-4964-8021-2bb9c4d390c3-data-volume\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.113873 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.113747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.113919 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.113870    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e4192f75-9684-4964-8021-2bb9c4d390c3-crio-socket\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.113961 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.113913    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvhdv\" (UniqueName: \"kubernetes.io/projected/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-api-access-tvhdv\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.114008 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.113976    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.215028 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.214994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4192f75-9684-4964-8021-2bb9c4d390c3-data-volume\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.215210 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.215045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.215210 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:16.215123    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 15:35:16.215210 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:16.215188    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm podName:e4192f75-9684-4964-8021-2bb9c4d390c3 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:16.7151711 +0000 UTC m=+60.260461754 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rgc2" (UID: "e4192f75-9684-4964-8021-2bb9c4d390c3") : configmap "kube-rbac-proxy" not found
Sep 18 15:35:16.215387 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.215221    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e4192f75-9684-4964-8021-2bb9c4d390c3-crio-socket\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.215387 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.215250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvhdv\" (UniqueName: \"kubernetes.io/projected/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-api-access-tvhdv\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.215387 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.215316    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.215516 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.215398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e4192f75-9684-4964-8021-2bb9c4d390c3-data-volume\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.215516 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.215469    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e4192f75-9684-4964-8021-2bb9c4d390c3-crio-socket\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.215516 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:16.215478    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:16.215610 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:16.215546    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls podName:e4192f75-9684-4964-8021-2bb9c4d390c3 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:16.715533263 +0000 UTC m=+60.260823914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rgc2" (UID: "e4192f75-9684-4964-8021-2bb9c4d390c3") : secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:16.241993 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.241929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvhdv\" (UniqueName: \"kubernetes.io/projected/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-api-access-tvhdv\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.638923 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.638850    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 15:35:16.639267 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.639249    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/0.log"
Sep 18 15:35:16.639311 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.639283    2561 generic.go:356] "Generic (PLEG): container finished" podID="577230bb-9bb3-42c4-9e6f-dda006b1552f" containerID="b41dd28ff21607be32d4d639f944f30155a2905695f70fdf570d3619d506ea07" exitCode=255
Sep 18 15:35:16.639367 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.639353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx" event={"ID":"577230bb-9bb3-42c4-9e6f-dda006b1552f","Type":"ContainerDied","Data":"b41dd28ff21607be32d4d639f944f30155a2905695f70fdf570d3619d506ea07"}
Sep 18 15:35:16.639415 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.639404    2561 scope.go:117] "RemoveContainer" containerID="81f475253c0f6c9ce47ce3607ada5d777f1b8a8a5d3ce74d3790204c21cb680f"
Sep 18 15:35:16.639643 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.639621    2561 scope.go:117] "RemoveContainer" containerID="b41dd28ff21607be32d4d639f944f30155a2905695f70fdf570d3619d506ea07"
Sep 18 15:35:16.639897 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:16.639851    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-db9cx_openshift-console-operator(577230bb-9bb3-42c4-9e6f-dda006b1552f)\"" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx" podUID="577230bb-9bb3-42c4-9e6f-dda006b1552f"
Sep 18 15:35:16.720519 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.720482    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.721198 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:16.720607    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 15:35:16.721198 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:16.720703    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm podName:e4192f75-9684-4964-8021-2bb9c4d390c3 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:17.720656776 +0000 UTC m=+61.265947445 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rgc2" (UID: "e4192f75-9684-4964-8021-2bb9c4d390c3") : configmap "kube-rbac-proxy" not found
Sep 18 15:35:16.721198 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:16.720891    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:16.721198 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:16.720988    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:16.721198 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:16.721063    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls podName:e4192f75-9684-4964-8021-2bb9c4d390c3 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:17.72103954 +0000 UTC m=+61.266330208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rgc2" (UID: "e4192f75-9684-4964-8021-2bb9c4d390c3") : secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:17.544593 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:17.544564    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wn6wq"
Sep 18 15:35:17.643795 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:17.643764    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 15:35:17.644149 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:17.644048    2561 scope.go:117] "RemoveContainer" containerID="b41dd28ff21607be32d4d639f944f30155a2905695f70fdf570d3619d506ea07"
Sep 18 15:35:17.644223 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:17.644205    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-db9cx_openshift-console-operator(577230bb-9bb3-42c4-9e6f-dda006b1552f)\"" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx" podUID="577230bb-9bb3-42c4-9e6f-dda006b1552f"
Sep 18 15:35:17.708692 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:17.708655    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-57tph_f2339bc3-3053-47db-b94c-dd9c926069a1/dns-node-resolver/0.log"
Sep 18 15:35:17.730528 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:17.730505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:17.730634 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:17.730618    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:17.730732 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:17.730705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:17.730853 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:17.730780    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls podName:e4192f75-9684-4964-8021-2bb9c4d390c3 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:19.730752318 +0000 UTC m=+63.276042984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rgc2" (UID: "e4192f75-9684-4964-8021-2bb9c4d390c3") : secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:17.730853 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:17.730794    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 15:35:17.730953 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:17.730899    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm podName:e4192f75-9684-4964-8021-2bb9c4d390c3 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:19.730887089 +0000 UTC m=+63.276177743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rgc2" (UID: "e4192f75-9684-4964-8021-2bb9c4d390c3") : configmap "kube-rbac-proxy" not found
Sep 18 15:35:18.035931 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:18.035892    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:18.036095 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:18.035959    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:18.646807 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:18.646782    2561 scope.go:117] "RemoveContainer" containerID="b41dd28ff21607be32d4d639f944f30155a2905695f70fdf570d3619d506ea07"
Sep 18 15:35:18.647166 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:18.646924    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-db9cx_openshift-console-operator(577230bb-9bb3-42c4-9e6f-dda006b1552f)\"" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx" podUID="577230bb-9bb3-42c4-9e6f-dda006b1552f"
Sep 18 15:35:18.908725 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:18.908637    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m7hxp_4f4df3e2-04a4-4534-8c8d-aba5d753f87a/node-ca/0.log"
Sep 18 15:35:19.743836 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:19.743798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:19.744196 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:19.743871    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:19.744196 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:19.743906    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 15:35:19.744196 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:19.743966    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm podName:e4192f75-9684-4964-8021-2bb9c4d390c3 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:23.743951234 +0000 UTC m=+67.289241886 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rgc2" (UID: "e4192f75-9684-4964-8021-2bb9c4d390c3") : configmap "kube-rbac-proxy" not found
Sep 18 15:35:19.744196 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:19.743966    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:19.744196 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:19.743992    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls podName:e4192f75-9684-4964-8021-2bb9c4d390c3 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:23.743986135 +0000 UTC m=+67.289276787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rgc2" (UID: "e4192f75-9684-4964-8021-2bb9c4d390c3") : secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:19.908891 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:19.908868    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rqkgz_84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1/serve-healthcheck-canary/0.log"
Sep 18 15:35:23.371480 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:23.371433    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77rbk\" (UID: \"d0d4b309-4771-4000-8bdd-70ee7489136b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"
Sep 18 15:35:23.373950 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:23.373926    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d0d4b309-4771-4000-8bdd-70ee7489136b-samples-operator-tls\") pod \"cluster-samples-operator-685d7d68b5-77rbk\" (UID: \"d0d4b309-4771-4000-8bdd-70ee7489136b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"
Sep 18 15:35:23.426064 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:23.426037    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vx52s\""
Sep 18 15:35:23.433248 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:23.433228    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"
Sep 18 15:35:23.472855 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:23.472823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:35:23.473037 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:23.472877    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:23.473037 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:23.472925    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:35:23.473037 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.472974    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 15:35:23.473037 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.472992    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbbbbbbb5-sqwnt: secret "image-registry-tls" not found
Sep 18 15:35:23.473037 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.473036    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 15:35:23.473234 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.473048    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls podName:b215b96a-f47b-4e8c-8df3-3c1aa336a57e nodeName:}" failed. No retries permitted until 2026-09-18 15:35:55.473034381 +0000 UTC m=+99.018325034 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls") pod "image-registry-cbbbbbbb5-sqwnt" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e") : secret "image-registry-tls" not found
Sep 18 15:35:23.473234 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.473037    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 18 15:35:23.473234 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.473101    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls podName:ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:55.473084729 +0000 UTC m=+99.018375395 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls") pod "dns-default-mmqzn" (UID: "ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407") : secret "dns-default-metrics-tls" not found
Sep 18 15:35:23.473234 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.473153    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls podName:52dadbd4-3e28-42d7-9d38-b4343c8c37ab nodeName:}" failed. No retries permitted until 2026-09-18 15:35:39.47313771 +0000 UTC m=+83.018428377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-czd8s" (UID: "52dadbd4-3e28-42d7-9d38-b4343c8c37ab") : secret "cluster-monitoring-operator-tls" not found
Sep 18 15:35:23.573405 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:23.573370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:23.573557 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:23.573524    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:23.573638 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.573553    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle podName:3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:39.573534733 +0000 UTC m=+83.118825384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle") pod "router-default-6bf6bd4c64-ppwrg" (UID: "3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436") : configmap references non-existent config key: service-ca.crt
Sep 18 15:35:23.573638 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.573616    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 15:35:23.573769 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.573696    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs podName:3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:39.57365789 +0000 UTC m=+83.118948565 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs") pod "router-default-6bf6bd4c64-ppwrg" (UID: "3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436") : secret "router-metrics-certs-default" not found
Sep 18 15:35:23.716464 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:23.716415    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk"]
Sep 18 15:35:23.774435 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:23.774407    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:23.774585 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.774518    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 15:35:23.774652 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.774594    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm podName:e4192f75-9684-4964-8021-2bb9c4d390c3 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:31.774574662 +0000 UTC m=+75.319865332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8rgc2" (UID: "e4192f75-9684-4964-8021-2bb9c4d390c3") : configmap "kube-rbac-proxy" not found
Sep 18 15:35:23.774652 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:23.774618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:23.774794 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.774776    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:23.774836 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:23.774826    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls podName:e4192f75-9684-4964-8021-2bb9c4d390c3 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:31.774814623 +0000 UTC m=+75.320105276 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rgc2" (UID: "e4192f75-9684-4964-8021-2bb9c4d390c3") : secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:24.077217 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.077136    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:35:24.077386 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.077279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:35:24.082718 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.082699    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 15:35:24.082816 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.082736    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 15:35:24.088233 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:24.088218    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 15:35:24.088300 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:24.088278    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs podName:e39d26a4-b893-4a51-b493-4d1b69476b2a nodeName:}" failed. No retries permitted until 2026-09-18 15:36:28.088260154 +0000 UTC m=+131.633550809 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs") pod "network-metrics-daemon-vpmt5" (UID: "e39d26a4-b893-4a51-b493-4d1b69476b2a") : secret "metrics-daemon-secret" not found
Sep 18 15:35:24.090282 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.090264    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea-original-pull-secret\") pod \"global-pull-secret-syncer-gjx2r\" (UID: \"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea\") " pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:35:24.178547 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.178514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmtr8\" (UniqueName: \"kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8\") pod \"network-check-target-v4ffn\" (UID: \"541880d8-884d-4644-9c35-3485805a5425\") " pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:35:24.180934 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.180909    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmtr8\" (UniqueName: \"kubernetes.io/projected/541880d8-884d-4644-9c35-3485805a5425-kube-api-access-nmtr8\") pod \"network-check-target-v4ffn\" (UID: \"541880d8-884d-4644-9c35-3485805a5425\") " pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:35:24.191650 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.191629    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nqjbs\""
Sep 18 15:35:24.199070 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.199054    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:35:24.292283 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.292256    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjx2r"
Sep 18 15:35:24.471760 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.471741    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v4ffn"]
Sep 18 15:35:24.473998 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:24.473967    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541880d8_884d_4644_9c35_3485805a5425.slice/crio-8fa886869c0c78766d9a105db08e91ceaca72417ba2753855bb1fd4b031d0607 WatchSource:0}: Error finding container 8fa886869c0c78766d9a105db08e91ceaca72417ba2753855bb1fd4b031d0607: Status 404 returned error can't find the container with id 8fa886869c0c78766d9a105db08e91ceaca72417ba2753855bb1fd4b031d0607
Sep 18 15:35:24.573786 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.573762    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gjx2r"]
Sep 18 15:35:24.576383 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:24.576359    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcedebd21_fe8d_4e66_a7b4_f3dcf76e9aea.slice/crio-111d7aba43114f6d997b70a2d03e2d296d6d4206893b012136976e1dbad3f558 WatchSource:0}: Error finding container 111d7aba43114f6d997b70a2d03e2d296d6d4206893b012136976e1dbad3f558: Status 404 returned error can't find the container with id 111d7aba43114f6d997b70a2d03e2d296d6d4206893b012136976e1dbad3f558
Sep 18 15:35:24.669170 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.669139    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk" event={"ID":"d0d4b309-4771-4000-8bdd-70ee7489136b","Type":"ContainerStarted","Data":"7f90b59c71b75c1ee62c4fd1429433da0053f06a40bccec00c30c2b81e658881"}
Sep 18 15:35:24.670315 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.670288    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gjx2r" event={"ID":"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea","Type":"ContainerStarted","Data":"111d7aba43114f6d997b70a2d03e2d296d6d4206893b012136976e1dbad3f558"}
Sep 18 15:35:24.671701 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.671680    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v4ffn" event={"ID":"541880d8-884d-4644-9c35-3485805a5425","Type":"ContainerStarted","Data":"8f56406ace4f01b5eba288006a63d4eecf21e5c9c9e3d143bb43fd5b1c504bb6"}
Sep 18 15:35:24.671803 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.671713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v4ffn" event={"ID":"541880d8-884d-4644-9c35-3485805a5425","Type":"ContainerStarted","Data":"8fa886869c0c78766d9a105db08e91ceaca72417ba2753855bb1fd4b031d0607"}
Sep 18 15:35:24.671868 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.671803    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:35:24.690155 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:24.690102    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v4ffn" podStartSLOduration=66.690089703 podStartE2EDuration="1m6.690089703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:24.689223033 +0000 UTC m=+68.234513705" watchObservedRunningTime="2026-09-18 15:35:24.690089703 +0000 UTC m=+68.235380398"
Sep 18 15:35:26.681963 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:26.681924    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk" event={"ID":"d0d4b309-4771-4000-8bdd-70ee7489136b","Type":"ContainerStarted","Data":"800998a2eb7c5559dce4b8a0c7b3c2ee63a8fc3cde468a83be10af7af3cb3f7d"}
Sep 18 15:35:26.682434 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:26.681980    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk" event={"ID":"d0d4b309-4771-4000-8bdd-70ee7489136b","Type":"ContainerStarted","Data":"f7e636312c766910d76f44d13dc757be6cf3ead7122ee2d81d93af5bd30d6807"}
Sep 18 15:35:26.702470 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:26.702425    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-685d7d68b5-77rbk" podStartSLOduration=17.545657805 podStartE2EDuration="19.702407576s" totalImagesPullingTime="2.156749771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:26.700588087 +0000 UTC m=+70.245878754" watchObservedRunningTime="2026-09-18 15:35:26.702407576 +0000 UTC m=+70.247698250"
Sep 18 15:35:29.692114 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:29.692076    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gjx2r" event={"ID":"cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea","Type":"ContainerStarted","Data":"7c61654fd5737cab40544ef46885b97d4d52589820e3240a825f3bc6853f745c"}
Sep 18 15:35:29.707506 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:29.707459    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gjx2r" podStartSLOduration=67.601235062 podStartE2EDuration="1m11.707445424s" totalImagesPullingTime="4.106210362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:29.706544775 +0000 UTC m=+73.251835448" watchObservedRunningTime="2026-09-18 15:35:29.707445424 +0000 UTC m=+73.252736098"
Sep 18 15:35:31.846150 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:31.846108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:31.846630 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:31.846241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:31.846630 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:31.846261    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:31.846630 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:31.846325    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls podName:e4192f75-9684-4964-8021-2bb9c4d390c3 nodeName:}" failed. No retries permitted until 2026-09-18 15:35:47.846311703 +0000 UTC m=+91.391602360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8rgc2" (UID: "e4192f75-9684-4964-8021-2bb9c4d390c3") : secret "insights-runtime-extractor-tls" not found
Sep 18 15:35:31.846897 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:31.846877    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e4192f75-9684-4964-8021-2bb9c4d390c3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:32.373878 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:32.373844    2561 scope.go:117] "RemoveContainer" containerID="b41dd28ff21607be32d4d639f944f30155a2905695f70fdf570d3619d506ea07"
Sep 18 15:35:32.707681 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:32.707634    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 15:35:32.707868 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:32.707757    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx" event={"ID":"577230bb-9bb3-42c4-9e6f-dda006b1552f","Type":"ContainerStarted","Data":"561efdc63f9f46cba652fb84cfbe7f83a74724980d46bc1795d22e49982041e4"}
Sep 18 15:35:32.708162 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:32.708142    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:32.730566 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:32.730510    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx" podStartSLOduration=19.55129062 podStartE2EDuration="25.73049104s" totalImagesPullingTime="6.17920042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:32.728419397 +0000 UTC m=+76.273710070" watchObservedRunningTime="2026-09-18 15:35:32.73049104 +0000 UTC m=+76.275781715"
Sep 18 15:35:32.997318 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:32.997242    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-db9cx"
Sep 18 15:35:39.509472 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:39.509342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:39.511685 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:39.511650    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/52dadbd4-3e28-42d7-9d38-b4343c8c37ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-czd8s\" (UID: \"52dadbd4-3e28-42d7-9d38-b4343c8c37ab\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:39.610348 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:39.610306    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:39.610490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:39.610439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:39.611065 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:39.611044    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-service-ca-bundle\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:39.612653 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:39.612633    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436-metrics-certs\") pod \"router-default-6bf6bd4c64-ppwrg\" (UID: \"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436\") " pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:39.717584 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:39.717553    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-psf8j\""
Sep 18 15:35:39.725159 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:39.725141    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"
Sep 18 15:35:39.855457 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:39.855430    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-9gvdh\""
Sep 18 15:35:39.863044 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:39.863019    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:39.991368 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:39.991296    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s"]
Sep 18 15:35:39.993743 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:39.993718    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52dadbd4_3e28_42d7_9d38_b4343c8c37ab.slice/crio-9ddb4bee5a8bd2c9867146f439d48039504b44a5f6bbbb5b8cad35a29e86c7de WatchSource:0}: Error finding container 9ddb4bee5a8bd2c9867146f439d48039504b44a5f6bbbb5b8cad35a29e86c7de: Status 404 returned error can't find the container with id 9ddb4bee5a8bd2c9867146f439d48039504b44a5f6bbbb5b8cad35a29e86c7de
Sep 18 15:35:40.131322 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.131301    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6bf6bd4c64-ppwrg"]
Sep 18 15:35:40.133338 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:40.133317    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbd12ae_c043_47ca_bcb5_c6ac2ecf1436.slice/crio-80dd4dc9cf6cd24b110f9b69cad5337cab6c0f3b7d56eca5228e67fbc3df779b WatchSource:0}: Error finding container 80dd4dc9cf6cd24b110f9b69cad5337cab6c0f3b7d56eca5228e67fbc3df779b: Status 404 returned error can't find the container with id 80dd4dc9cf6cd24b110f9b69cad5337cab6c0f3b7d56eca5228e67fbc3df779b
Sep 18 15:35:40.698646 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.698614    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-hznvw"]
Sep 18 15:35:40.700681 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.700640    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-hznvw"
Sep 18 15:35:40.705719 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.705695    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 15:35:40.705814 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.705773    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 15:35:40.705887 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.705810    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-tx7th\""
Sep 18 15:35:40.730119 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.730096    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-hznvw"]
Sep 18 15:35:40.739618 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.739046    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg" event={"ID":"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436","Type":"ContainerStarted","Data":"208cce95ce49ad6542f83619cb2de513dac0126ccbdd0139c46fe8271bbd693b"}
Sep 18 15:35:40.739618 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.739096    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg" event={"ID":"3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436","Type":"ContainerStarted","Data":"80dd4dc9cf6cd24b110f9b69cad5337cab6c0f3b7d56eca5228e67fbc3df779b"}
Sep 18 15:35:40.742066 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.742041    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s" event={"ID":"52dadbd4-3e28-42d7-9d38-b4343c8c37ab","Type":"ContainerStarted","Data":"9ddb4bee5a8bd2c9867146f439d48039504b44a5f6bbbb5b8cad35a29e86c7de"}
Sep 18 15:35:40.802979 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.802936    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg" podStartSLOduration=33.802920776 podStartE2EDuration="33.802920776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:40.802043183 +0000 UTC m=+84.347333856" watchObservedRunningTime="2026-09-18 15:35:40.802920776 +0000 UTC m=+84.348211449"
Sep 18 15:35:40.822272 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.821622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkmk4\" (UniqueName: \"kubernetes.io/projected/60f6b3de-1cb4-47e1-9d46-1c1292690318-kube-api-access-zkmk4\") pod \"downloads-6594f4c75-hznvw\" (UID: \"60f6b3de-1cb4-47e1-9d46-1c1292690318\") " pod="openshift-console/downloads-6594f4c75-hznvw"
Sep 18 15:35:40.863804 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.863777    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:40.866408 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.866388    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:40.922405 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.922382    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkmk4\" (UniqueName: \"kubernetes.io/projected/60f6b3de-1cb4-47e1-9d46-1c1292690318-kube-api-access-zkmk4\") pod \"downloads-6594f4c75-hznvw\" (UID: \"60f6b3de-1cb4-47e1-9d46-1c1292690318\") " pod="openshift-console/downloads-6594f4c75-hznvw"
Sep 18 15:35:40.936961 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:40.936906    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkmk4\" (UniqueName: \"kubernetes.io/projected/60f6b3de-1cb4-47e1-9d46-1c1292690318-kube-api-access-zkmk4\") pod \"downloads-6594f4c75-hznvw\" (UID: \"60f6b3de-1cb4-47e1-9d46-1c1292690318\") " pod="openshift-console/downloads-6594f4c75-hznvw"
Sep 18 15:35:41.012060 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:41.011980    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-hznvw"
Sep 18 15:35:41.292051 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:41.291984    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-hznvw"]
Sep 18 15:35:41.746914 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:41.746888    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:41.748157 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:41.748135    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6bf6bd4c64-ppwrg"
Sep 18 15:35:41.830859 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:41.830823    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f6b3de_1cb4_47e1_9d46_1c1292690318.slice/crio-ca20ad3b6fd67750986f9f9f2d2adca6d9f755eff029073c44bf63b9418a19e9 WatchSource:0}: Error finding container ca20ad3b6fd67750986f9f9f2d2adca6d9f755eff029073c44bf63b9418a19e9: Status 404 returned error can't find the container with id ca20ad3b6fd67750986f9f9f2d2adca6d9f755eff029073c44bf63b9418a19e9
Sep 18 15:35:41.890354 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:41.890320    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-554fd97b7-kt2x8"]
Sep 18 15:35:41.904731 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:41.904711    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:41.912206 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:41.912190    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 15:35:41.913021 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:41.913004    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 15:35:41.913356 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:41.913336    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8fq62\""
Sep 18 15:35:41.913435 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:41.913353    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 15:35:41.913792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:41.913772    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 15:35:41.913953 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:41.913933    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 15:35:41.915244 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:41.915224    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-554fd97b7-kt2x8"]
Sep 18 15:35:42.033238 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.033172    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-serving-cert\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.033238 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.033204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-oauth-serving-cert\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.033238 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.033228    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c9p2\" (UniqueName: \"kubernetes.io/projected/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-kube-api-access-6c9p2\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.033428 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.033276    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-oauth-config\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.033428 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.033396    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-service-ca\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.033492 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.033428    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-config\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.134400 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.134365    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-service-ca\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.134566 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.134412    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-config\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.134631 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.134561    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-serving-cert\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.134631 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.134587    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-oauth-serving-cert\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.134631 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.134607    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c9p2\" (UniqueName: \"kubernetes.io/projected/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-kube-api-access-6c9p2\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.134817 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.134633    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-oauth-config\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.135054 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.135034    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-service-ca\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.135143 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.135121    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-config\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.135258 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.135243    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-oauth-serving-cert\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.136998 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.136982    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-oauth-config\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.137053 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.136984    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-serving-cert\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.142802 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.142783    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c9p2\" (UniqueName: \"kubernetes.io/projected/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-kube-api-access-6c9p2\") pod \"console-554fd97b7-kt2x8\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") " pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.214415 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.214392    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:42.509893 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.509856    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-554fd97b7-kt2x8"]
Sep 18 15:35:42.513272 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:42.513241    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b562e98_9d5a_48f4_9c0b_ea50fdd087db.slice/crio-f8384c50f2aeb6113e2856726c47c53885f81789403108bf2127ba6fd060d589 WatchSource:0}: Error finding container f8384c50f2aeb6113e2856726c47c53885f81789403108bf2127ba6fd060d589: Status 404 returned error can't find the container with id f8384c50f2aeb6113e2856726c47c53885f81789403108bf2127ba6fd060d589
Sep 18 15:35:42.751691 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.751640    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554fd97b7-kt2x8" event={"ID":"1b562e98-9d5a-48f4-9c0b-ea50fdd087db","Type":"ContainerStarted","Data":"f8384c50f2aeb6113e2856726c47c53885f81789403108bf2127ba6fd060d589"}
Sep 18 15:35:42.753795 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.753759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s" event={"ID":"52dadbd4-3e28-42d7-9d38-b4343c8c37ab","Type":"ContainerStarted","Data":"18f332198a1f97913f2dcf180511603f0d3b454b50ee949812fff6b204b0a3f4"}
Sep 18 15:35:42.755183 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.755152    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-hznvw" event={"ID":"60f6b3de-1cb4-47e1-9d46-1c1292690318","Type":"ContainerStarted","Data":"ca20ad3b6fd67750986f9f9f2d2adca6d9f755eff029073c44bf63b9418a19e9"}
Sep 18 15:35:42.774051 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:42.773957    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-czd8s" podStartSLOduration=33.593489305 podStartE2EDuration="35.773937178s" totalImagesPullingTime="2.180447873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:42.772824658 +0000 UTC m=+86.318115332" watchObservedRunningTime="2026-09-18 15:35:42.773937178 +0000 UTC m=+86.319227853"
Sep 18 15:35:47.778398 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:47.778364    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554fd97b7-kt2x8" event={"ID":"1b562e98-9d5a-48f4-9c0b-ea50fdd087db","Type":"ContainerStarted","Data":"2bb17ac2f8cbb11e5724de2b0128db3270825d34c11be1ae30e424603515ae93"}
Sep 18 15:35:47.795869 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:47.795820    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-554fd97b7-kt2x8" podStartSLOduration=1.973900065 podStartE2EDuration="6.795804503s" totalImagesPullingTime="4.821904438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:47.79515773 +0000 UTC m=+91.340448404" watchObservedRunningTime="2026-09-18 15:35:47.795804503 +0000 UTC m=+91.341095174"
Sep 18 15:35:47.891240 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:47.891208    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:47.893812 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:47.893785    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e4192f75-9684-4964-8021-2bb9c4d390c3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8rgc2\" (UID: \"e4192f75-9684-4964-8021-2bb9c4d390c3\") " pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:48.127040 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:48.126961    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c72vd\""
Sep 18 15:35:48.135020 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:48.134993    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8rgc2"
Sep 18 15:35:48.413810 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:48.413736    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8rgc2"]
Sep 18 15:35:48.418619 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:48.418459    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4192f75_9684_4964_8021_2bb9c4d390c3.slice/crio-00660902d2c297cf75fe3de650471933d5dfe529c970bdf43c6d9546e6d14dd9 WatchSource:0}: Error finding container 00660902d2c297cf75fe3de650471933d5dfe529c970bdf43c6d9546e6d14dd9: Status 404 returned error can't find the container with id 00660902d2c297cf75fe3de650471933d5dfe529c970bdf43c6d9546e6d14dd9
Sep 18 15:35:48.783869 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:48.783829    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8rgc2" event={"ID":"e4192f75-9684-4964-8021-2bb9c4d390c3","Type":"ContainerStarted","Data":"8970adcaa6b78d390eaf462c6b14b0470543e0b33d536e0ab7cae4259091482c"}
Sep 18 15:35:48.784299 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:48.783896    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8rgc2" event={"ID":"e4192f75-9684-4964-8021-2bb9c4d390c3","Type":"ContainerStarted","Data":"00660902d2c297cf75fe3de650471933d5dfe529c970bdf43c6d9546e6d14dd9"}
Sep 18 15:35:49.789535 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:49.789498    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8rgc2" event={"ID":"e4192f75-9684-4964-8021-2bb9c4d390c3","Type":"ContainerStarted","Data":"a631e39cdae407d43dea11c006d56c996e0be5abb3c5616e85dcf11ac1882ee3"}
Sep 18 15:35:51.801142 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:51.801104    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8rgc2" event={"ID":"e4192f75-9684-4964-8021-2bb9c4d390c3","Type":"ContainerStarted","Data":"4358a3d2858729cf1ea4649252f9668de818f0abc18a1c244fbeb36a7d913869"}
Sep 18 15:35:51.822744 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:51.822686    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8rgc2" podStartSLOduration=34.244276974 podStartE2EDuration="36.822655266s" totalImagesPullingTime="2.578378292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 15:35:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:51.820485966 +0000 UTC m=+95.365776631" watchObservedRunningTime="2026-09-18 15:35:51.822655266 +0000 UTC m=+95.367945943"
Sep 18 15:35:52.215077 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:52.215040    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:52.215276 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:52.215115    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:52.221559 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:52.221535    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:52.810213 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:52.810183    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:35:55.565011 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:55.564979    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:35:55.565421 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:55.565026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:35:55.567634 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:55.567606    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407-metrics-tls\") pod \"dns-default-mmqzn\" (UID: \"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407\") " pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:35:55.567857 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:55.567834    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls\") pod \"image-registry-cbbbbbbb5-sqwnt\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") " pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:35:55.679408 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:55.679378    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v4ffn"
Sep 18 15:35:55.819643 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:55.819558    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7tpfr\""
Sep 18 15:35:55.827748 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:55.827721    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:35:55.860817 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:55.860793    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f9wls\""
Sep 18 15:35:55.868468 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:55.868441    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:35:57.982431 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:57.982394    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-554fd97b7-kt2x8"]
Sep 18 15:35:58.394065 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.393988    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-948xd"]
Sep 18 15:35:58.400291 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.400270    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.402949 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.402925    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kpxdd\""
Sep 18 15:35:58.404266 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.404243    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 15:35:58.404266 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.404263    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 15:35:58.404424 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.404277    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 15:35:58.404424 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.404317    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 15:35:58.491861 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.491828    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-root\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.491861 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.491864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.492099 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.491889    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-sys\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.492099 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.491933    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-accelerators-collector-config\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.492193 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.492106    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-tls\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.492193 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.492141    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-wtmp\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.492193 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.492174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-metrics-client-ca\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.492309 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.492217    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr4pd\" (UniqueName: \"kubernetes.io/projected/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-kube-api-access-vr4pd\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.492309 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.492243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-textfile\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593041 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593003    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-tls\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593215 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593050    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-wtmp\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593215 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593086    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-metrics-client-ca\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593215 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593128    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr4pd\" (UniqueName: \"kubernetes.io/projected/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-kube-api-access-vr4pd\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593215 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593155    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-textfile\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593215 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593189    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-root\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593215 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593522 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593241    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-sys\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593522 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593265    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-accelerators-collector-config\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593522 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593266    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-wtmp\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593522 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593338    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-root\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593522 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593342    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-sys\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593793 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593655    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-textfile\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.593910 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593884    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-metrics-client-ca\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.594694 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.593940    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-accelerators-collector-config\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.595873 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.595852    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-tls\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.595972 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.595910    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.601478 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.601458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr4pd\" (UniqueName: \"kubernetes.io/projected/469cc9fd-0676-4be8-8d6f-c5031f06e0c2-kube-api-access-vr4pd\") pod \"node-exporter-948xd\" (UID: \"469cc9fd-0676-4be8-8d6f-c5031f06e0c2\") " pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.711284 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.711250    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-948xd"
Sep 18 15:35:58.822749 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:58.822699    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469cc9fd_0676_4be8_8d6f_c5031f06e0c2.slice/crio-df7e2a8a355c1ccdd70b542129728c5f4f4534e6bca0ef44dda67e131a587458 WatchSource:0}: Error finding container df7e2a8a355c1ccdd70b542129728c5f4f4534e6bca0ef44dda67e131a587458: Status 404 returned error can't find the container with id df7e2a8a355c1ccdd70b542129728c5f4f4534e6bca0ef44dda67e131a587458
Sep 18 15:35:58.831054 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:58.831023    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-948xd" event={"ID":"469cc9fd-0676-4be8-8d6f-c5031f06e0c2","Type":"ContainerStarted","Data":"df7e2a8a355c1ccdd70b542129728c5f4f4534e6bca0ef44dda67e131a587458"}
Sep 18 15:35:59.089197 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.089161    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mmqzn"]
Sep 18 15:35:59.092154 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.092129    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"]
Sep 18 15:35:59.092818 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:59.092792    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9f2a0c_fd5b_4e1e_8c64_8131e8b89407.slice/crio-cdc72ea41759d1c28b30eb78bb487c5e765c19edc57e0437317631210369e802 WatchSource:0}: Error finding container cdc72ea41759d1c28b30eb78bb487c5e765c19edc57e0437317631210369e802: Status 404 returned error can't find the container with id cdc72ea41759d1c28b30eb78bb487c5e765c19edc57e0437317631210369e802
Sep 18 15:35:59.094943 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:35:59.094914    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb215b96a_f47b_4e8c_8df3_3c1aa336a57e.slice/crio-b4a40edbedb59479f0a6fcb66a041d4bfb15d92fff47284d8216343b447a0e37 WatchSource:0}: Error finding container b4a40edbedb59479f0a6fcb66a041d4bfb15d92fff47284d8216343b447a0e37: Status 404 returned error can't find the container with id b4a40edbedb59479f0a6fcb66a041d4bfb15d92fff47284d8216343b447a0e37
Sep 18 15:35:59.316737 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.315695    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 15:35:59.320148 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.320119    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.322696 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.322646    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 15:35:59.323049 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.323025    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 15:35:59.323368 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.323351    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 15:35:59.323673 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.323634    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 15:35:59.323876 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.323857    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 15:35:59.324083 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.324062    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 15:35:59.324765 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.324559    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 15:35:59.324765 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.324559    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 15:35:59.324765 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.324580    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-s865b\""
Sep 18 15:35:59.325068 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.325045    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 15:35:59.336642 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.336559    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 15:35:59.401182 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.401146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9349358a-28e2-4369-a8cc-4b8215982bad-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.401362 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.401206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.401362 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.401246    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9349358a-28e2-4369-a8cc-4b8215982bad-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.401362 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.401284    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnm4d\" (UniqueName: \"kubernetes.io/projected/9349358a-28e2-4369-a8cc-4b8215982bad-kube-api-access-wnm4d\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.401362 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.401341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.401578 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.401381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-config-volume\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.401578 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.401428    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-web-config\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.401578 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.401452    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9349358a-28e2-4369-a8cc-4b8215982bad-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.401578 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.401488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9349358a-28e2-4369-a8cc-4b8215982bad-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.401578 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.401520    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9349358a-28e2-4369-a8cc-4b8215982bad-config-out\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.401578 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.401545    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.401895 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.401584    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.401895 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.401612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.503131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.502896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9349358a-28e2-4369-a8cc-4b8215982bad-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.503131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.502955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnm4d\" (UniqueName: \"kubernetes.io/projected/9349358a-28e2-4369-a8cc-4b8215982bad-kube-api-access-wnm4d\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.503131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.502997    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.503131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.503037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-config-volume\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.503131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.503070    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-web-config\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.503131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.503094    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9349358a-28e2-4369-a8cc-4b8215982bad-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.503131 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.503117    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9349358a-28e2-4369-a8cc-4b8215982bad-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.503614 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.503147    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9349358a-28e2-4369-a8cc-4b8215982bad-config-out\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.503614 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.503172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.503614 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.503198    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.503614 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.503229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.503614 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.503288    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9349358a-28e2-4369-a8cc-4b8215982bad-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.503614 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.503312    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.504148 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.504115    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9349358a-28e2-4369-a8cc-4b8215982bad-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.504259 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:35:59.504244    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9349358a-28e2-4369-a8cc-4b8215982bad-alertmanager-trusted-ca-bundle podName:9349358a-28e2-4369-a8cc-4b8215982bad nodeName:}" failed. No retries permitted until 2026-09-18 15:36:00.00422333 +0000 UTC m=+103.549513999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/9349358a-28e2-4369-a8cc-4b8215982bad-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "9349358a-28e2-4369-a8cc-4b8215982bad") : configmap references non-existent config key: ca-bundle.crt
Sep 18 15:35:59.507372 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.506953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9349358a-28e2-4369-a8cc-4b8215982bad-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.507372 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.506975    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.509138 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.508934    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.511086 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.510983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9349358a-28e2-4369-a8cc-4b8215982bad-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.512047 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.511973    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-web-config\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.512144 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.512110    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-config-volume\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.512716 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.512325    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9349358a-28e2-4369-a8cc-4b8215982bad-config-out\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.512716 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.512654    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.513544 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.513518    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.514526 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.514496    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9349358a-28e2-4369-a8cc-4b8215982bad-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.515373 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.515264    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnm4d\" (UniqueName: \"kubernetes.io/projected/9349358a-28e2-4369-a8cc-4b8215982bad-kube-api-access-wnm4d\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:35:59.839835 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.839601    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-hznvw" event={"ID":"60f6b3de-1cb4-47e1-9d46-1c1292690318","Type":"ContainerStarted","Data":"c077d804441006b15495c25dc848fdd53c6cac5fa3103c93fce9a119b9950306"}
Sep 18 15:35:59.842991 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.842799    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-hznvw"
Sep 18 15:35:59.845712 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.845646    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mmqzn" event={"ID":"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407","Type":"ContainerStarted","Data":"cdc72ea41759d1c28b30eb78bb487c5e765c19edc57e0437317631210369e802"}
Sep 18 15:35:59.847980 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.847960    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-hznvw"
Sep 18 15:35:59.848440 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.848302    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt" event={"ID":"b215b96a-f47b-4e8c-8df3-3c1aa336a57e","Type":"ContainerStarted","Data":"4d47096b4c4895bb3728e845cc79e4be36237f4afa97108520415de1b25f85f0"}
Sep 18 15:35:59.848440 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.848343    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt" event={"ID":"b215b96a-f47b-4e8c-8df3-3c1aa336a57e","Type":"ContainerStarted","Data":"b4a40edbedb59479f0a6fcb66a041d4bfb15d92fff47284d8216343b447a0e37"}
Sep 18 15:35:59.848714 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.848651    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:35:59.860182 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.860092    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-hznvw" podStartSLOduration=2.773862481 podStartE2EDuration="19.860073562s" totalImagesPullingTime="17.086211081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:59.858814184 +0000 UTC m=+103.404104858" watchObservedRunningTime="2026-09-18 15:35:59.860073562 +0000 UTC m=+103.405364234"
Sep 18 15:35:59.908752 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:35:59.907244    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt" podStartSLOduration=101.907224315 podStartE2EDuration="1m41.907224315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:35:59.905881862 +0000 UTC m=+103.451172538" watchObservedRunningTime="2026-09-18 15:35:59.907224315 +0000 UTC m=+103.452514987"
Sep 18 15:36:00.009021 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:00.008980    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9349358a-28e2-4369-a8cc-4b8215982bad-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:36:00.010514 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:00.010482    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9349358a-28e2-4369-a8cc-4b8215982bad-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9349358a-28e2-4369-a8cc-4b8215982bad\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:36:00.233278 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:00.233038    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 15:36:00.618084 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:00.617986    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 15:36:00.717241 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:36:00.717176    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9349358a_28e2_4369_a8cc_4b8215982bad.slice/crio-0fb2f7f14a35bf8bac3b47ef92d1cb4fa8d548efacdb80d09225075f3d1c4891 WatchSource:0}: Error finding container 0fb2f7f14a35bf8bac3b47ef92d1cb4fa8d548efacdb80d09225075f3d1c4891: Status 404 returned error can't find the container with id 0fb2f7f14a35bf8bac3b47ef92d1cb4fa8d548efacdb80d09225075f3d1c4891
Sep 18 15:36:00.854966 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:00.854922    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9349358a-28e2-4369-a8cc-4b8215982bad","Type":"ContainerStarted","Data":"0fb2f7f14a35bf8bac3b47ef92d1cb4fa8d548efacdb80d09225075f3d1c4891"}
Sep 18 15:36:01.306646 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.306009    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"]
Sep 18 15:36:01.333507 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.333467    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"]
Sep 18 15:36:01.333657 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.333615    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.346574 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.346552    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3iobbs7meva4u\""
Sep 18 15:36:01.347270 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.347250    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 15:36:01.347765 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.347747    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 15:36:01.348000 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.347981    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 15:36:01.348105 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.347995    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-gvtt2\""
Sep 18 15:36:01.348635 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.348478    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 15:36:01.348759 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.348742    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 15:36:01.424413 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.424376    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.424596 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.424434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.424596 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.424463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e2b419c-6e64-475b-b85e-e6e297d8ef57-metrics-client-ca\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.424596 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.424537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt5ft\" (UniqueName: \"kubernetes.io/projected/3e2b419c-6e64-475b-b85e-e6e297d8ef57-kube-api-access-xt5ft\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.424782 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.424597    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-grpc-tls\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.424782 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.424675    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.424782 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.424739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.424782 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.424780    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-tls\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.525917 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.525882    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.526050 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.525935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.526050 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.525961    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e2b419c-6e64-475b-b85e-e6e297d8ef57-metrics-client-ca\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.526050 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.525998    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt5ft\" (UniqueName: \"kubernetes.io/projected/3e2b419c-6e64-475b-b85e-e6e297d8ef57-kube-api-access-xt5ft\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.526050 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.526039    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-grpc-tls\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.526263 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.526095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.526263 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.526145    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.526263 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.526175    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-tls\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.527146 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.527092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e2b419c-6e64-475b-b85e-e6e297d8ef57-metrics-client-ca\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.529337 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.529291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.529464 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.529435    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.529464 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.529458    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-grpc-tls\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.530218 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.530142    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-tls\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.530218 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.530168    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.531375 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.531349    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3e2b419c-6e64-475b-b85e-e6e297d8ef57-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.545022 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.545004    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt5ft\" (UniqueName: \"kubernetes.io/projected/3e2b419c-6e64-475b-b85e-e6e297d8ef57-kube-api-access-xt5ft\") pod \"thanos-querier-55fdc8b964-jlv2l\" (UID: \"3e2b419c-6e64-475b-b85e-e6e297d8ef57\") " pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.646295 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.646174    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:01.862900 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.862861    2561 generic.go:356] "Generic (PLEG): container finished" podID="469cc9fd-0676-4be8-8d6f-c5031f06e0c2" containerID="4ae7e595649a888d6cfea70ad9df54d7672525e6b82fcb8dcc36aa32cc18491f" exitCode=0
Sep 18 15:36:01.863061 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.862917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-948xd" event={"ID":"469cc9fd-0676-4be8-8d6f-c5031f06e0c2","Type":"ContainerDied","Data":"4ae7e595649a888d6cfea70ad9df54d7672525e6b82fcb8dcc36aa32cc18491f"}
Sep 18 15:36:01.938604 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:01.938571    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"]
Sep 18 15:36:01.942679 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:36:01.942633    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2b419c_6e64_475b_b85e_e6e297d8ef57.slice/crio-8bbecf7d8b19495f7edf5421bebf27a26c74ed8c97a7a77c350be3f6559da64c WatchSource:0}: Error finding container 8bbecf7d8b19495f7edf5421bebf27a26c74ed8c97a7a77c350be3f6559da64c: Status 404 returned error can't find the container with id 8bbecf7d8b19495f7edf5421bebf27a26c74ed8c97a7a77c350be3f6559da64c
Sep 18 15:36:02.872105 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:02.872073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-948xd" event={"ID":"469cc9fd-0676-4be8-8d6f-c5031f06e0c2","Type":"ContainerStarted","Data":"f2924311c71882b0fb585515c55edd058afce4a69e905d583434f197331ef961"}
Sep 18 15:36:02.872550 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:02.872166    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-948xd" event={"ID":"469cc9fd-0676-4be8-8d6f-c5031f06e0c2","Type":"ContainerStarted","Data":"37217a3fdb5dcb89fd1b8a731d922749fdbe7586704bae1cfd57ce2c32ca1262"}
Sep 18 15:36:02.875693 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:02.875612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mmqzn" event={"ID":"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407","Type":"ContainerStarted","Data":"5d988754926349d22ffb65809d924bf2e6598bdb40b7d92fdbceb3d1c6d86346"}
Sep 18 15:36:02.875862 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:02.875838    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mmqzn" event={"ID":"ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407","Type":"ContainerStarted","Data":"ab34670dae55c89890434068a6fdac7c6956ec6179c04f654083c52a32d0803c"}
Sep 18 15:36:02.876400 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:02.876374    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:36:02.878217 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:02.878126    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l" event={"ID":"3e2b419c-6e64-475b-b85e-e6e297d8ef57","Type":"ContainerStarted","Data":"8bbecf7d8b19495f7edf5421bebf27a26c74ed8c97a7a77c350be3f6559da64c"}
Sep 18 15:36:02.900852 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:02.900799    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-948xd" podStartSLOduration=2.891303979 podStartE2EDuration="4.900782803s" totalImagesPullingTime="1.918934456s" totalInitContainerRuntime="90.544368ms" isStatefulPod=true podCreationTimestamp="2026-09-18 15:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:36:02.900746346 +0000 UTC m=+106.446037021" watchObservedRunningTime="2026-09-18 15:36:02.900782803 +0000 UTC m=+106.446073479"
Sep 18 15:36:02.928003 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:02.927951    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mmqzn" podStartSLOduration=69.070455676 podStartE2EDuration="1m11.927931791s" totalImagesPullingTime="2.857476115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:36:02.926437749 +0000 UTC m=+106.471728424" watchObservedRunningTime="2026-09-18 15:36:02.927931791 +0000 UTC m=+106.473222464"
Sep 18 15:36:03.034326 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.034287    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58984b4b77-pnpzw"]
Sep 18 15:36:03.063752 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.063641    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58984b4b77-pnpzw"]
Sep 18 15:36:03.063982 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.063782    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.073135 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.073110    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 18 15:36:03.119656 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.119614    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"]
Sep 18 15:36:03.146035 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.145967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-trusted-ca-bundle\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.146035 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.146024    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-oauth-config\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.146301 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.146064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-config\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.146301 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.146082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t72rb\" (UniqueName: \"kubernetes.io/projected/554c0c77-fdb6-421d-8134-fce196cb4bdb-kube-api-access-t72rb\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.146301 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.146114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-oauth-serving-cert\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.146301 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.146143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-serving-cert\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.146301 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.146206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-service-ca\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.246915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.246881    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-trusted-ca-bundle\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.247080 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.246952    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-oauth-config\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.247080 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.246995    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-config\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.247080 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.247022    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t72rb\" (UniqueName: \"kubernetes.io/projected/554c0c77-fdb6-421d-8134-fce196cb4bdb-kube-api-access-t72rb\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.247080 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.247056    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-oauth-serving-cert\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.247296 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.247089    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-serving-cert\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.247296 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.247154    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-service-ca\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.248990 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.247847    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-trusted-ca-bundle\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.248990 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.247876    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-service-ca\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.248990 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.248394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-oauth-serving-cert\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.248990 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.248952    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-config\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.250375 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.250352    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-oauth-config\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.253814 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.253788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-serving-cert\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.261100 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.261053    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t72rb\" (UniqueName: \"kubernetes.io/projected/554c0c77-fdb6-421d-8134-fce196cb4bdb-kube-api-access-t72rb\") pod \"console-58984b4b77-pnpzw\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") " pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.379305 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.379274    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:03.692987 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.692949    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58984b4b77-pnpzw"]
Sep 18 15:36:03.885153 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.885107    2561 generic.go:356] "Generic (PLEG): container finished" podID="9349358a-28e2-4369-a8cc-4b8215982bad" containerID="672a3843970e8b682b22b7568a2c3fde74374053c0b179b0867e4d1a64e4ac44" exitCode=0
Sep 18 15:36:03.885618 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:03.885496    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9349358a-28e2-4369-a8cc-4b8215982bad","Type":"ContainerDied","Data":"672a3843970e8b682b22b7568a2c3fde74374053c0b179b0867e4d1a64e4ac44"}
Sep 18 15:36:04.317345 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:36:04.317305    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554c0c77_fdb6_421d_8134_fce196cb4bdb.slice/crio-06bc26a5fc6ff2e82b14947b05bce0892515a67240d1433026bfef7ce748fbf6 WatchSource:0}: Error finding container 06bc26a5fc6ff2e82b14947b05bce0892515a67240d1433026bfef7ce748fbf6: Status 404 returned error can't find the container with id 06bc26a5fc6ff2e82b14947b05bce0892515a67240d1433026bfef7ce748fbf6
Sep 18 15:36:04.895039 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:04.894927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l" event={"ID":"3e2b419c-6e64-475b-b85e-e6e297d8ef57","Type":"ContainerStarted","Data":"d9e7c30a5fa8d8ed31f6ee5e0ea87ae8db6a8f68dee97637687eaad3c1d04d6d"}
Sep 18 15:36:04.895415 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:04.895048    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l" event={"ID":"3e2b419c-6e64-475b-b85e-e6e297d8ef57","Type":"ContainerStarted","Data":"33c4ac08ad2e5c9f6d07c6c6678479b0777921ccbfd10290071a0271614a6002"}
Sep 18 15:36:04.898721 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:04.898379    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58984b4b77-pnpzw" event={"ID":"554c0c77-fdb6-421d-8134-fce196cb4bdb","Type":"ContainerStarted","Data":"d0b4297a219308268d92608bed23a6af618ec600a326034e04e7e88b794c1da0"}
Sep 18 15:36:04.898721 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:04.898425    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58984b4b77-pnpzw" event={"ID":"554c0c77-fdb6-421d-8134-fce196cb4bdb","Type":"ContainerStarted","Data":"06bc26a5fc6ff2e82b14947b05bce0892515a67240d1433026bfef7ce748fbf6"}
Sep 18 15:36:05.907001 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:05.906960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l" event={"ID":"3e2b419c-6e64-475b-b85e-e6e297d8ef57","Type":"ContainerStarted","Data":"161dd442b0c3d4035e84755932774f4dde6bca90946fe8837a5236f6e4290568"}
Sep 18 15:36:06.914458 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:06.914384    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9349358a-28e2-4369-a8cc-4b8215982bad","Type":"ContainerStarted","Data":"ec4d81dc53c61d695e5f502c4fdeb14fcaac322cdc5da301ae21d4713b47669c"}
Sep 18 15:36:06.914864 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:06.914468    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9349358a-28e2-4369-a8cc-4b8215982bad","Type":"ContainerStarted","Data":"a3538b9b9ac1ed59df86658ea1488bc31480b3a68b472feff007bfd82301b2fc"}
Sep 18 15:36:06.914864 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:06.914507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9349358a-28e2-4369-a8cc-4b8215982bad","Type":"ContainerStarted","Data":"57ab0de5ed3d956dec685b297fcf603929a693fa9da274707e96b6dd98227023"}
Sep 18 15:36:07.924915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:07.924880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9349358a-28e2-4369-a8cc-4b8215982bad","Type":"ContainerStarted","Data":"f6ba065245371e1de34faffaadf1aa86fb01325c4a20caa765f7d652721f6888"}
Sep 18 15:36:07.925349 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:07.925001    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9349358a-28e2-4369-a8cc-4b8215982bad","Type":"ContainerStarted","Data":"0ec1ddfe6b6d459f12ee7831c32c7dcf60754e8ccdc67ac0c32f68700ae9d11f"}
Sep 18 15:36:07.925349 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:07.925031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9349358a-28e2-4369-a8cc-4b8215982bad","Type":"ContainerStarted","Data":"6471ba6e6ff738632a19bc5adf1ea093856ba18a3c555060d8449fcc128aa434"}
Sep 18 15:36:07.927740 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:07.927710    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l" event={"ID":"3e2b419c-6e64-475b-b85e-e6e297d8ef57","Type":"ContainerStarted","Data":"84ecb138478435773a2c735f61541996101c67478995b26e8a5f79d07e1bdda4"}
Sep 18 15:36:07.927855 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:07.927777    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l" event={"ID":"3e2b419c-6e64-475b-b85e-e6e297d8ef57","Type":"ContainerStarted","Data":"342186b14a88f8ec8e2ff40fdbbde605da081a5e08042a5081577edf845435cc"}
Sep 18 15:36:07.927855 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:07.927819    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l" event={"ID":"3e2b419c-6e64-475b-b85e-e6e297d8ef57","Type":"ContainerStarted","Data":"278b3473ca4d29792e2f37827b2d0269eddae9fe4d0a7891368c409bba7bbbe4"}
Sep 18 15:36:07.928242 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:07.927864    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:07.953517 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:07.953478    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.108398825 podStartE2EDuration="8.953465019s" totalImagesPullingTime="4.822197636s" totalInitContainerRuntime="22.868558ms" isStatefulPod=false podCreationTimestamp="2026-09-18 15:35:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:36:07.951981177 +0000 UTC m=+111.497271850" watchObservedRunningTime="2026-09-18 15:36:07.953465019 +0000 UTC m=+111.498755692"
Sep 18 15:36:07.954050 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:07.954023    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58984b4b77-pnpzw" podStartSLOduration=4.95401545 podStartE2EDuration="4.95401545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:36:04.944863514 +0000 UTC m=+108.490154192" watchObservedRunningTime="2026-09-18 15:36:07.95401545 +0000 UTC m=+111.499306124"
Sep 18 15:36:07.974676 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:07.974615    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l" podStartSLOduration=2.240346067 podStartE2EDuration="6.97459627s" totalImagesPullingTime="4.734250203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:36:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:36:07.974008254 +0000 UTC m=+111.519298928" watchObservedRunningTime="2026-09-18 15:36:07.97459627 +0000 UTC m=+111.519886946"
Sep 18 15:36:13.379512 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:13.379476    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:13.380016 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:13.379536    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:13.384223 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:13.384203    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:13.900722 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:13.900693    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mmqzn"
Sep 18 15:36:13.937648 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:13.937618    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55fdc8b964-jlv2l"
Sep 18 15:36:13.955384 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:13.955354    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:36:23.004239 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.004195    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-554fd97b7-kt2x8" podUID="1b562e98-9d5a-48f4-9c0b-ea50fdd087db" containerName="console" containerID="cri-o://2bb17ac2f8cbb11e5724de2b0128db3270825d34c11be1ae30e424603515ae93" gracePeriod=15
Sep 18 15:36:23.125608 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.125581    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:36:23.275801 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.275741    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:36:23.422777 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.422746    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-oauth-config\") pod \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") "
Sep 18 15:36:23.422968 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.422784    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-oauth-serving-cert\") pod \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") "
Sep 18 15:36:23.422968 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.422818    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-serving-cert\") pod \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") "
Sep 18 15:36:23.422968 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.422852    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-service-ca\") pod \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") "
Sep 18 15:36:23.422968 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.422875    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c9p2\" (UniqueName: \"kubernetes.io/projected/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-kube-api-access-6c9p2\") pod \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") "
Sep 18 15:36:23.422968 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.422904    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-config\") pod \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\" (UID: \"1b562e98-9d5a-48f4-9c0b-ea50fdd087db\") "
Sep 18 15:36:23.423276 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.423242    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1b562e98-9d5a-48f4-9c0b-ea50fdd087db" (UID: "1b562e98-9d5a-48f4-9c0b-ea50fdd087db"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:36:23.423276 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.423264    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-service-ca" (OuterVolumeSpecName: "service-ca") pod "1b562e98-9d5a-48f4-9c0b-ea50fdd087db" (UID: "1b562e98-9d5a-48f4-9c0b-ea50fdd087db"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:36:23.423742 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.423719    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-config" (OuterVolumeSpecName: "console-config") pod "1b562e98-9d5a-48f4-9c0b-ea50fdd087db" (UID: "1b562e98-9d5a-48f4-9c0b-ea50fdd087db"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:36:23.425169 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.425141    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1b562e98-9d5a-48f4-9c0b-ea50fdd087db" (UID: "1b562e98-9d5a-48f4-9c0b-ea50fdd087db"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:36:23.425251 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.425211    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1b562e98-9d5a-48f4-9c0b-ea50fdd087db" (UID: "1b562e98-9d5a-48f4-9c0b-ea50fdd087db"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:36:23.425251 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.425232    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-kube-api-access-6c9p2" (OuterVolumeSpecName: "kube-api-access-6c9p2") pod "1b562e98-9d5a-48f4-9c0b-ea50fdd087db" (UID: "1b562e98-9d5a-48f4-9c0b-ea50fdd087db"). InnerVolumeSpecName "kube-api-access-6c9p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:36:23.523540 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.523519    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-oauth-config\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:23.523540 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.523538    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-oauth-serving-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:23.523732 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.523547    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-serving-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:23.523732 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.523557    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-service-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:23.523732 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.523566    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c9p2\" (UniqueName: \"kubernetes.io/projected/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-kube-api-access-6c9p2\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:23.523732 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.523575    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1b562e98-9d5a-48f4-9c0b-ea50fdd087db-console-config\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:23.991274 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.991237    2561 generic.go:356] "Generic (PLEG): container finished" podID="1b562e98-9d5a-48f4-9c0b-ea50fdd087db" containerID="2bb17ac2f8cbb11e5724de2b0128db3270825d34c11be1ae30e424603515ae93" exitCode=0
Sep 18 15:36:23.991496 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.991295    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554fd97b7-kt2x8"
Sep 18 15:36:23.991496 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.991327    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554fd97b7-kt2x8" event={"ID":"1b562e98-9d5a-48f4-9c0b-ea50fdd087db","Type":"ContainerDied","Data":"2bb17ac2f8cbb11e5724de2b0128db3270825d34c11be1ae30e424603515ae93"}
Sep 18 15:36:23.991496 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.991400    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554fd97b7-kt2x8" event={"ID":"1b562e98-9d5a-48f4-9c0b-ea50fdd087db","Type":"ContainerDied","Data":"f8384c50f2aeb6113e2856726c47c53885f81789403108bf2127ba6fd060d589"}
Sep 18 15:36:23.991496 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:23.991422    2561 scope.go:117] "RemoveContainer" containerID="2bb17ac2f8cbb11e5724de2b0128db3270825d34c11be1ae30e424603515ae93"
Sep 18 15:36:24.000274 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:24.000257    2561 scope.go:117] "RemoveContainer" containerID="2bb17ac2f8cbb11e5724de2b0128db3270825d34c11be1ae30e424603515ae93"
Sep 18 15:36:24.000573 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:36:24.000550    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bb17ac2f8cbb11e5724de2b0128db3270825d34c11be1ae30e424603515ae93\": container with ID starting with 2bb17ac2f8cbb11e5724de2b0128db3270825d34c11be1ae30e424603515ae93 not found: ID does not exist" containerID="2bb17ac2f8cbb11e5724de2b0128db3270825d34c11be1ae30e424603515ae93"
Sep 18 15:36:24.000610 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:24.000580    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bb17ac2f8cbb11e5724de2b0128db3270825d34c11be1ae30e424603515ae93"} err="failed to get container status \"2bb17ac2f8cbb11e5724de2b0128db3270825d34c11be1ae30e424603515ae93\": rpc error: code = NotFound desc = could not find container \"2bb17ac2f8cbb11e5724de2b0128db3270825d34c11be1ae30e424603515ae93\": container with ID starting with 2bb17ac2f8cbb11e5724de2b0128db3270825d34c11be1ae30e424603515ae93 not found: ID does not exist"
Sep 18 15:36:24.013325 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:24.013303    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-554fd97b7-kt2x8"]
Sep 18 15:36:24.016876 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:24.016857    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-554fd97b7-kt2x8"]
Sep 18 15:36:24.381388 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:24.381311    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b562e98-9d5a-48f4-9c0b-ea50fdd087db" path="/var/lib/kubelet/pods/1b562e98-9d5a-48f4-9c0b-ea50fdd087db/volumes"
Sep 18 15:36:26.000813 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:26.000780    2561 generic.go:356] "Generic (PLEG): container finished" podID="42f4c511-6c8c-45f6-a86d-bb6f5d204022" containerID="ad6a6723fbf2e0ba5e19657af9e727159b3e886cb19973b0f2d2a4a4e48a72ac" exitCode=0
Sep 18 15:36:26.001221 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:26.000851    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-2965g" event={"ID":"42f4c511-6c8c-45f6-a86d-bb6f5d204022","Type":"ContainerDied","Data":"ad6a6723fbf2e0ba5e19657af9e727159b3e886cb19973b0f2d2a4a4e48a72ac"}
Sep 18 15:36:26.001285 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:26.001245    2561 scope.go:117] "RemoveContainer" containerID="ad6a6723fbf2e0ba5e19657af9e727159b3e886cb19973b0f2d2a4a4e48a72ac"
Sep 18 15:36:26.002425 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:26.002404    2561 generic.go:356] "Generic (PLEG): container finished" podID="2def30a4-9a64-4048-8d1f-90b6ae7f3075" containerID="3dee9ec1bd8dc3c7eed8991bcb49029d990d75ad081aa5cc343a95316384ce78" exitCode=0
Sep 18 15:36:26.002425 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:26.002431    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9" event={"ID":"2def30a4-9a64-4048-8d1f-90b6ae7f3075","Type":"ContainerDied","Data":"3dee9ec1bd8dc3c7eed8991bcb49029d990d75ad081aa5cc343a95316384ce78"}
Sep 18 15:36:26.002714 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:26.002700    2561 scope.go:117] "RemoveContainer" containerID="3dee9ec1bd8dc3c7eed8991bcb49029d990d75ad081aa5cc343a95316384ce78"
Sep 18 15:36:27.007782 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:27.007749    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-2965g" event={"ID":"42f4c511-6c8c-45f6-a86d-bb6f5d204022","Type":"ContainerStarted","Data":"7bd8fec4f501c06706859e76c1ece3e7e70442f3aa7924309fdaa6456c200f8a"}
Sep 18 15:36:27.009537 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:27.009514    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-vfzz9" event={"ID":"2def30a4-9a64-4048-8d1f-90b6ae7f3075","Type":"ContainerStarted","Data":"57d824963f73f95e8905f19f4766c41c21edecba18e2c9c167031f659cf4068a"}
Sep 18 15:36:28.143623 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.143577    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt" podUID="b215b96a-f47b-4e8c-8df3-3c1aa336a57e" containerName="registry" containerID="cri-o://4d47096b4c4895bb3728e845cc79e4be36237f4afa97108520415de1b25f85f0" gracePeriod=30
Sep 18 15:36:28.162354 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.162329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:36:28.164552 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.164530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e39d26a4-b893-4a51-b493-4d1b69476b2a-metrics-certs\") pod \"network-metrics-daemon-vpmt5\" (UID: \"e39d26a4-b893-4a51-b493-4d1b69476b2a\") " pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:36:28.199485 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.199462    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c572m\""
Sep 18 15:36:28.207546 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.207529    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vpmt5"
Sep 18 15:36:28.396078 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.396025    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:36:28.496705 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.496679    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vpmt5"]
Sep 18 15:36:28.499130 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:36:28.499103    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39d26a4_b893_4a51_b493_4d1b69476b2a.slice/crio-bd04b1350bd259c35514e507a733cf7cce6c8abebf6a731752d62aaf84219d35 WatchSource:0}: Error finding container bd04b1350bd259c35514e507a733cf7cce6c8abebf6a731752d62aaf84219d35: Status 404 returned error can't find the container with id bd04b1350bd259c35514e507a733cf7cce6c8abebf6a731752d62aaf84219d35
Sep 18 15:36:28.565684 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.565637    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-ca-trust-extracted\") pod \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") "
Sep 18 15:36:28.565805 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.565721    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-certificates\") pod \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") "
Sep 18 15:36:28.565805 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.565763    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-bound-sa-token\") pod \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") "
Sep 18 15:36:28.565805 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.565796    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-trusted-ca\") pod \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") "
Sep 18 15:36:28.566027 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.565839    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-installation-pull-secrets\") pod \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") "
Sep 18 15:36:28.566027 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.565871    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls\") pod \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") "
Sep 18 15:36:28.566027 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.565906    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-image-registry-private-configuration\") pod \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") "
Sep 18 15:36:28.566027 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.565950    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkh5m\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-kube-api-access-zkh5m\") pod \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\" (UID: \"b215b96a-f47b-4e8c-8df3-3c1aa336a57e\") "
Sep 18 15:36:28.566214 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.566130    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b215b96a-f47b-4e8c-8df3-3c1aa336a57e" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:36:28.566271 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.566236    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-certificates\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:28.566868 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.566831    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b215b96a-f47b-4e8c-8df3-3c1aa336a57e" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:36:28.568609 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.568581    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b215b96a-f47b-4e8c-8df3-3c1aa336a57e" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:36:28.568755 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.568733    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b215b96a-f47b-4e8c-8df3-3c1aa336a57e" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:36:28.568755 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.568738    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b215b96a-f47b-4e8c-8df3-3c1aa336a57e" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:36:28.568862 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.568823    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b215b96a-f47b-4e8c-8df3-3c1aa336a57e" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:36:28.568862 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.568822    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-kube-api-access-zkh5m" (OuterVolumeSpecName: "kube-api-access-zkh5m") pod "b215b96a-f47b-4e8c-8df3-3c1aa336a57e" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e"). InnerVolumeSpecName "kube-api-access-zkh5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:36:28.573861 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.573840    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b215b96a-f47b-4e8c-8df3-3c1aa336a57e" (UID: "b215b96a-f47b-4e8c-8df3-3c1aa336a57e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:36:28.666628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.666570    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-bound-sa-token\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:28.666628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.666590    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-trusted-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:28.666628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.666599    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-installation-pull-secrets\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:28.666628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.666608    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-registry-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:28.666628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.666618    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-image-registry-private-configuration\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:28.666628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.666627    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkh5m\" (UniqueName: \"kubernetes.io/projected/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-kube-api-access-zkh5m\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:28.666910 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:28.666635    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b215b96a-f47b-4e8c-8df3-3c1aa336a57e-ca-trust-extracted\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:36:29.020059 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:29.020026    2561 generic.go:356] "Generic (PLEG): container finished" podID="b215b96a-f47b-4e8c-8df3-3c1aa336a57e" containerID="4d47096b4c4895bb3728e845cc79e4be36237f4afa97108520415de1b25f85f0" exitCode=0
Sep 18 15:36:29.020234 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:29.020070    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"
Sep 18 15:36:29.020234 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:29.020114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt" event={"ID":"b215b96a-f47b-4e8c-8df3-3c1aa336a57e","Type":"ContainerDied","Data":"4d47096b4c4895bb3728e845cc79e4be36237f4afa97108520415de1b25f85f0"}
Sep 18 15:36:29.020234 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:29.020177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cbbbbbbb5-sqwnt" event={"ID":"b215b96a-f47b-4e8c-8df3-3c1aa336a57e","Type":"ContainerDied","Data":"b4a40edbedb59479f0a6fcb66a041d4bfb15d92fff47284d8216343b447a0e37"}
Sep 18 15:36:29.020234 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:29.020209    2561 scope.go:117] "RemoveContainer" containerID="4d47096b4c4895bb3728e845cc79e4be36237f4afa97108520415de1b25f85f0"
Sep 18 15:36:29.021493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:29.021470    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vpmt5" event={"ID":"e39d26a4-b893-4a51-b493-4d1b69476b2a","Type":"ContainerStarted","Data":"bd04b1350bd259c35514e507a733cf7cce6c8abebf6a731752d62aaf84219d35"}
Sep 18 15:36:29.030843 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:29.030828    2561 scope.go:117] "RemoveContainer" containerID="4d47096b4c4895bb3728e845cc79e4be36237f4afa97108520415de1b25f85f0"
Sep 18 15:36:29.031118 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:36:29.031092    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d47096b4c4895bb3728e845cc79e4be36237f4afa97108520415de1b25f85f0\": container with ID starting with 4d47096b4c4895bb3728e845cc79e4be36237f4afa97108520415de1b25f85f0 not found: ID does not exist" containerID="4d47096b4c4895bb3728e845cc79e4be36237f4afa97108520415de1b25f85f0"
Sep 18 15:36:29.031187 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:29.031130    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d47096b4c4895bb3728e845cc79e4be36237f4afa97108520415de1b25f85f0"} err="failed to get container status \"4d47096b4c4895bb3728e845cc79e4be36237f4afa97108520415de1b25f85f0\": rpc error: code = NotFound desc = could not find container \"4d47096b4c4895bb3728e845cc79e4be36237f4afa97108520415de1b25f85f0\": container with ID starting with 4d47096b4c4895bb3728e845cc79e4be36237f4afa97108520415de1b25f85f0 not found: ID does not exist"
Sep 18 15:36:29.045230 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:29.045208    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"]
Sep 18 15:36:29.049026 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:29.049002    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cbbbbbbb5-sqwnt"]
Sep 18 15:36:30.029529 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:30.029485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vpmt5" event={"ID":"e39d26a4-b893-4a51-b493-4d1b69476b2a","Type":"ContainerStarted","Data":"e7a53637661b6168ddf8ff079085812b6b4c2d6856a17330c2d02ca84580ee29"}
Sep 18 15:36:30.030049 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:30.029571    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vpmt5" event={"ID":"e39d26a4-b893-4a51-b493-4d1b69476b2a","Type":"ContainerStarted","Data":"719d7976d8d7c0471bcecc28821d4e4c38039f06c3e30fd5893fb1b7987ab05f"}
Sep 18 15:36:30.048415 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:30.048361    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vpmt5" podStartSLOduration=131.017968382 podStartE2EDuration="2m12.04834652s" totalImagesPullingTime="1.030378138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:34:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:36:30.046859573 +0000 UTC m=+133.592150260" watchObservedRunningTime="2026-09-18 15:36:30.04834652 +0000 UTC m=+133.593637194"
Sep 18 15:36:30.381008 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:30.380914    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b215b96a-f47b-4e8c-8df3-3c1aa336a57e" path="/var/lib/kubelet/pods/b215b96a-f47b-4e8c-8df3-3c1aa336a57e/volumes"
Sep 18 15:36:46.101818 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:46.101782    2561 generic.go:356] "Generic (PLEG): container finished" podID="32803062-157c-4605-8474-a32715b72352" containerID="9768dc93c04f9dc48a5d19aadb94442b717f3062cc17161e434299c0aaec2f86" exitCode=0
Sep 18 15:36:46.102158 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:46.101856    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc" event={"ID":"32803062-157c-4605-8474-a32715b72352","Type":"ContainerDied","Data":"9768dc93c04f9dc48a5d19aadb94442b717f3062cc17161e434299c0aaec2f86"}
Sep 18 15:36:46.102219 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:46.102197    2561 scope.go:117] "RemoveContainer" containerID="9768dc93c04f9dc48a5d19aadb94442b717f3062cc17161e434299c0aaec2f86"
Sep 18 15:36:47.107146 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:36:47.107114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-xn2fc" event={"ID":"32803062-157c-4605-8474-a32715b72352","Type":"ContainerStarted","Data":"6f2916af96f98fae92b02ec31e069499b5af989b06cf89ca313d7c344518d383"}
Sep 18 15:37:22.721674 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.721620    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56b848874f-pl85h"]
Sep 18 15:37:22.722058 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.721949    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b215b96a-f47b-4e8c-8df3-3c1aa336a57e" containerName="registry"
Sep 18 15:37:22.722058 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.721961    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b215b96a-f47b-4e8c-8df3-3c1aa336a57e" containerName="registry"
Sep 18 15:37:22.722058 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.721970    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b562e98-9d5a-48f4-9c0b-ea50fdd087db" containerName="console"
Sep 18 15:37:22.722058 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.721975    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b562e98-9d5a-48f4-9c0b-ea50fdd087db" containerName="console"
Sep 18 15:37:22.722058 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.722025    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b562e98-9d5a-48f4-9c0b-ea50fdd087db" containerName="console"
Sep 18 15:37:22.722058 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.722034    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b215b96a-f47b-4e8c-8df3-3c1aa336a57e" containerName="registry"
Sep 18 15:37:22.724582 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.724566    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.727375 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.727355    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jj4n8\""
Sep 18 15:37:22.727484 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.727373    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 18 15:37:22.727619 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.727604    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 18 15:37:22.727792 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.727779    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 18 15:37:22.727988 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.727960    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 18 15:37:22.728102 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.727966    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 18 15:37:22.732537 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.732517    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 18 15:37:22.742509 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.742482    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56b848874f-pl85h"]
Sep 18 15:37:22.815141 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.815112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54c6c283-2719-42f1-bfb1-009ede0820d0-serving-certs-ca-bundle\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.815141 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.815144    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/54c6c283-2719-42f1-bfb1-009ede0820d0-federate-client-tls\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.815348 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.815163    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54c6c283-2719-42f1-bfb1-009ede0820d0-metrics-client-ca\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.815348 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.815179    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfftt\" (UniqueName: \"kubernetes.io/projected/54c6c283-2719-42f1-bfb1-009ede0820d0-kube-api-access-nfftt\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.815348 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.815201    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/54c6c283-2719-42f1-bfb1-009ede0820d0-secret-telemeter-client\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.815348 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.815287    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54c6c283-2719-42f1-bfb1-009ede0820d0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.815348 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.815332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/54c6c283-2719-42f1-bfb1-009ede0820d0-telemeter-client-tls\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.815560 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.815399    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54c6c283-2719-42f1-bfb1-009ede0820d0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.916805 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.916778    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54c6c283-2719-42f1-bfb1-009ede0820d0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.916957 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.916826    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/54c6c283-2719-42f1-bfb1-009ede0820d0-telemeter-client-tls\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.916957 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.916863    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54c6c283-2719-42f1-bfb1-009ede0820d0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.916957 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.916888    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54c6c283-2719-42f1-bfb1-009ede0820d0-serving-certs-ca-bundle\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.916957 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.916910    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/54c6c283-2719-42f1-bfb1-009ede0820d0-federate-client-tls\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.916957 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.916935    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54c6c283-2719-42f1-bfb1-009ede0820d0-metrics-client-ca\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.917205 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.916959    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfftt\" (UniqueName: \"kubernetes.io/projected/54c6c283-2719-42f1-bfb1-009ede0820d0-kube-api-access-nfftt\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.917205 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.917093    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/54c6c283-2719-42f1-bfb1-009ede0820d0-secret-telemeter-client\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.917652 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.917627    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54c6c283-2719-42f1-bfb1-009ede0820d0-serving-certs-ca-bundle\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.917821 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.917771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54c6c283-2719-42f1-bfb1-009ede0820d0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.918296 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.918267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54c6c283-2719-42f1-bfb1-009ede0820d0-metrics-client-ca\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.919535 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.919511    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/54c6c283-2719-42f1-bfb1-009ede0820d0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.919807 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.919680    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/54c6c283-2719-42f1-bfb1-009ede0820d0-federate-client-tls\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.919807 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.919697    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/54c6c283-2719-42f1-bfb1-009ede0820d0-secret-telemeter-client\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.919899 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.919819    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/54c6c283-2719-42f1-bfb1-009ede0820d0-telemeter-client-tls\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:22.925443 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:22.925423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfftt\" (UniqueName: \"kubernetes.io/projected/54c6c283-2719-42f1-bfb1-009ede0820d0-kube-api-access-nfftt\") pod \"telemeter-client-56b848874f-pl85h\" (UID: \"54c6c283-2719-42f1-bfb1-009ede0820d0\") " pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:23.034513 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:23.034434    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56b848874f-pl85h"
Sep 18 15:37:23.310626 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:23.310496    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56b848874f-pl85h"]
Sep 18 15:37:23.312463 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:37:23.312437    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c6c283_2719_42f1_bfb1_009ede0820d0.slice/crio-84d22fdc30ad6b73000e23f9d380d76cbda973a9b509d221d0a64f184a9cbfc7 WatchSource:0}: Error finding container 84d22fdc30ad6b73000e23f9d380d76cbda973a9b509d221d0a64f184a9cbfc7: Status 404 returned error can't find the container with id 84d22fdc30ad6b73000e23f9d380d76cbda973a9b509d221d0a64f184a9cbfc7
Sep 18 15:37:24.258678 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:24.258620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56b848874f-pl85h" event={"ID":"54c6c283-2719-42f1-bfb1-009ede0820d0","Type":"ContainerStarted","Data":"84d22fdc30ad6b73000e23f9d380d76cbda973a9b509d221d0a64f184a9cbfc7"}
Sep 18 15:37:25.266237 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:25.266204    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56b848874f-pl85h" event={"ID":"54c6c283-2719-42f1-bfb1-009ede0820d0","Type":"ContainerStarted","Data":"589a871e6da4ed6b68d22fd6c4724460053a97f6ab0783221f0aa4f052eee293"}
Sep 18 15:37:25.266577 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:25.266261    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56b848874f-pl85h" event={"ID":"54c6c283-2719-42f1-bfb1-009ede0820d0","Type":"ContainerStarted","Data":"66666994c5571a07c4ae54679f53af53e2de4e31271a7392366ad14aa05beca8"}
Sep 18 15:37:25.266577 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:25.266280    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56b848874f-pl85h" event={"ID":"54c6c283-2719-42f1-bfb1-009ede0820d0","Type":"ContainerStarted","Data":"59faa52f4b0233831b1bf4ff5845ccdc8ceccffa2a63fd181002477c606477ac"}
Sep 18 15:37:25.292603 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:25.292557    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56b848874f-pl85h" podStartSLOduration=1.54515447 podStartE2EDuration="3.292541998s" totalImagesPullingTime="1.747387528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:37:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:37:25.292065875 +0000 UTC m=+188.837356549" watchObservedRunningTime="2026-09-18 15:37:25.292541998 +0000 UTC m=+188.837832671"
Sep 18 15:37:33.434517 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.434484    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54dfdb646c-bd42n"]
Sep 18 15:37:33.436767 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.436748    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.449595 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.449569    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54dfdb646c-bd42n"]
Sep 18 15:37:33.497937 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.497907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-serving-cert\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.498088 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.497955    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-config\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.498088 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.497982    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-service-ca\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.498088 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.498009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22tng\" (UniqueName: \"kubernetes.io/projected/b71954b1-f3c0-451a-a734-b6f5c392a8cf-kube-api-access-22tng\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.498088 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.498032    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-oauth-serving-cert\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.498235 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.498086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-trusted-ca-bundle\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.498235 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.498209    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-oauth-config\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.599112 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.599081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-config\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.599112 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.599116    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-service-ca\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.599314 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.599139    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22tng\" (UniqueName: \"kubernetes.io/projected/b71954b1-f3c0-451a-a734-b6f5c392a8cf-kube-api-access-22tng\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.599314 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.599162    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-oauth-serving-cert\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.599314 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.599183    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-trusted-ca-bundle\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.599314 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.599220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-oauth-config\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.599314 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.599244    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-serving-cert\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.600004 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.599979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-service-ca\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.600110 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.600013    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-oauth-serving-cert\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.600110 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.599979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-config\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.600438 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.600414    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-trusted-ca-bundle\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.601779 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.601760    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-oauth-config\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.601874 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.601856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-serving-cert\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.607511 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.607483    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22tng\" (UniqueName: \"kubernetes.io/projected/b71954b1-f3c0-451a-a734-b6f5c392a8cf-kube-api-access-22tng\") pod \"console-54dfdb646c-bd42n\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") " pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:33.745902 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:33.745827    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:34.016057 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:34.015977    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54dfdb646c-bd42n"]
Sep 18 15:37:34.019136 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:37:34.019109    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71954b1_f3c0_451a_a734_b6f5c392a8cf.slice/crio-90290fe70338c34323d475d748e57849328b229230a42f8e834c7f13c4a8ce9d WatchSource:0}: Error finding container 90290fe70338c34323d475d748e57849328b229230a42f8e834c7f13c4a8ce9d: Status 404 returned error can't find the container with id 90290fe70338c34323d475d748e57849328b229230a42f8e834c7f13c4a8ce9d
Sep 18 15:37:34.304975 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:34.304891    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dfdb646c-bd42n" event={"ID":"b71954b1-f3c0-451a-a734-b6f5c392a8cf","Type":"ContainerStarted","Data":"4bc43c7f6949cf03758f3b8d4c0ed3ec1704863759ba24b1cdf6b13fb00fe2f6"}
Sep 18 15:37:34.304975 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:34.304938    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dfdb646c-bd42n" event={"ID":"b71954b1-f3c0-451a-a734-b6f5c392a8cf","Type":"ContainerStarted","Data":"90290fe70338c34323d475d748e57849328b229230a42f8e834c7f13c4a8ce9d"}
Sep 18 15:37:34.323120 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:34.323074    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54dfdb646c-bd42n" podStartSLOduration=1.323058765 podStartE2EDuration="1.323058765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:37:34.322076009 +0000 UTC m=+197.867366684" watchObservedRunningTime="2026-09-18 15:37:34.323058765 +0000 UTC m=+197.868349439"
Sep 18 15:37:43.746129 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:43.746092    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:43.746530 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:43.746142    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:43.750944 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:43.750921    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:44.349214 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:44.349184    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:37:44.397535 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:37:44.397498    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58984b4b77-pnpzw"]
Sep 18 15:38:09.419401 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.419345    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58984b4b77-pnpzw" podUID="554c0c77-fdb6-421d-8134-fce196cb4bdb" containerName="console" containerID="cri-o://d0b4297a219308268d92608bed23a6af618ec600a326034e04e7e88b794c1da0" gracePeriod=15
Sep 18 15:38:09.662932 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.662910    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:38:09.812042 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.811955    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-oauth-serving-cert\") pod \"554c0c77-fdb6-421d-8134-fce196cb4bdb\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") "
Sep 18 15:38:09.812042 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.811990    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-serving-cert\") pod \"554c0c77-fdb6-421d-8134-fce196cb4bdb\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") "
Sep 18 15:38:09.812042 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.812031    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-oauth-config\") pod \"554c0c77-fdb6-421d-8134-fce196cb4bdb\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") "
Sep 18 15:38:09.812042 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.812046    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-config\") pod \"554c0c77-fdb6-421d-8134-fce196cb4bdb\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") "
Sep 18 15:38:09.812413 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.812063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t72rb\" (UniqueName: \"kubernetes.io/projected/554c0c77-fdb6-421d-8134-fce196cb4bdb-kube-api-access-t72rb\") pod \"554c0c77-fdb6-421d-8134-fce196cb4bdb\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") "
Sep 18 15:38:09.812413 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.812109    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-trusted-ca-bundle\") pod \"554c0c77-fdb6-421d-8134-fce196cb4bdb\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") "
Sep 18 15:38:09.812413 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.812165    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-service-ca\") pod \"554c0c77-fdb6-421d-8134-fce196cb4bdb\" (UID: \"554c0c77-fdb6-421d-8134-fce196cb4bdb\") "
Sep 18 15:38:09.812560 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.812453    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-config" (OuterVolumeSpecName: "console-config") pod "554c0c77-fdb6-421d-8134-fce196cb4bdb" (UID: "554c0c77-fdb6-421d-8134-fce196cb4bdb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:38:09.812560 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.812466    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "554c0c77-fdb6-421d-8134-fce196cb4bdb" (UID: "554c0c77-fdb6-421d-8134-fce196cb4bdb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:38:09.812655 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.812623    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "554c0c77-fdb6-421d-8134-fce196cb4bdb" (UID: "554c0c77-fdb6-421d-8134-fce196cb4bdb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:38:09.812733 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.812697    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-service-ca" (OuterVolumeSpecName: "service-ca") pod "554c0c77-fdb6-421d-8134-fce196cb4bdb" (UID: "554c0c77-fdb6-421d-8134-fce196cb4bdb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:38:09.814355 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.814328    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "554c0c77-fdb6-421d-8134-fce196cb4bdb" (UID: "554c0c77-fdb6-421d-8134-fce196cb4bdb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:38:09.814355 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.814345    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "554c0c77-fdb6-421d-8134-fce196cb4bdb" (UID: "554c0c77-fdb6-421d-8134-fce196cb4bdb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:38:09.814497 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.814417    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554c0c77-fdb6-421d-8134-fce196cb4bdb-kube-api-access-t72rb" (OuterVolumeSpecName: "kube-api-access-t72rb") pod "554c0c77-fdb6-421d-8134-fce196cb4bdb" (UID: "554c0c77-fdb6-421d-8134-fce196cb4bdb"). InnerVolumeSpecName "kube-api-access-t72rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:38:09.913616 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.913581    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-oauth-config\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:38:09.913616 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.913608    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-config\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:38:09.913616 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.913618    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t72rb\" (UniqueName: \"kubernetes.io/projected/554c0c77-fdb6-421d-8134-fce196cb4bdb-kube-api-access-t72rb\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:38:09.913862 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.913626    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-trusted-ca-bundle\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:38:09.913862 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.913637    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-service-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:38:09.913862 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.913645    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/554c0c77-fdb6-421d-8134-fce196cb4bdb-oauth-serving-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:38:09.913862 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:09.913653    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/554c0c77-fdb6-421d-8134-fce196cb4bdb-console-serving-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:38:10.451419 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:10.451385    2561 generic.go:356] "Generic (PLEG): container finished" podID="554c0c77-fdb6-421d-8134-fce196cb4bdb" containerID="d0b4297a219308268d92608bed23a6af618ec600a326034e04e7e88b794c1da0" exitCode=0
Sep 18 15:38:10.451790 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:10.451431    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58984b4b77-pnpzw"
Sep 18 15:38:10.451790 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:10.451468    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58984b4b77-pnpzw" event={"ID":"554c0c77-fdb6-421d-8134-fce196cb4bdb","Type":"ContainerDied","Data":"d0b4297a219308268d92608bed23a6af618ec600a326034e04e7e88b794c1da0"}
Sep 18 15:38:10.451790 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:10.451515    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58984b4b77-pnpzw" event={"ID":"554c0c77-fdb6-421d-8134-fce196cb4bdb","Type":"ContainerDied","Data":"06bc26a5fc6ff2e82b14947b05bce0892515a67240d1433026bfef7ce748fbf6"}
Sep 18 15:38:10.451790 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:10.451537    2561 scope.go:117] "RemoveContainer" containerID="d0b4297a219308268d92608bed23a6af618ec600a326034e04e7e88b794c1da0"
Sep 18 15:38:10.459645 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:10.459629    2561 scope.go:117] "RemoveContainer" containerID="d0b4297a219308268d92608bed23a6af618ec600a326034e04e7e88b794c1da0"
Sep 18 15:38:10.459949 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:38:10.459930    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b4297a219308268d92608bed23a6af618ec600a326034e04e7e88b794c1da0\": container with ID starting with d0b4297a219308268d92608bed23a6af618ec600a326034e04e7e88b794c1da0 not found: ID does not exist" containerID="d0b4297a219308268d92608bed23a6af618ec600a326034e04e7e88b794c1da0"
Sep 18 15:38:10.459996 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:10.459956    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b4297a219308268d92608bed23a6af618ec600a326034e04e7e88b794c1da0"} err="failed to get container status \"d0b4297a219308268d92608bed23a6af618ec600a326034e04e7e88b794c1da0\": rpc error: code = NotFound desc = could not find container \"d0b4297a219308268d92608bed23a6af618ec600a326034e04e7e88b794c1da0\": container with ID starting with d0b4297a219308268d92608bed23a6af618ec600a326034e04e7e88b794c1da0 not found: ID does not exist"
Sep 18 15:38:10.468299 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:10.468277    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58984b4b77-pnpzw"]
Sep 18 15:38:10.471699 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:10.471680    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58984b4b77-pnpzw"]
Sep 18 15:38:12.378408 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:38:12.378376    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="554c0c77-fdb6-421d-8134-fce196cb4bdb" path="/var/lib/kubelet/pods/554c0c77-fdb6-421d-8134-fce196cb4bdb/volumes"
Sep 18 15:39:18.270949 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:39:18.270923    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 15:39:18.280963 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:39:18.280937    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 15:39:18.286159 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:39:18.286134    2561 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 15:40:23.452156 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.452078    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5z52m"]
Sep 18 15:40:23.452612 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.452408    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="554c0c77-fdb6-421d-8134-fce196cb4bdb" containerName="console"
Sep 18 15:40:23.452612 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.452419    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="554c0c77-fdb6-421d-8134-fce196cb4bdb" containerName="console"
Sep 18 15:40:23.452612 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.452480    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="554c0c77-fdb6-421d-8134-fce196cb4bdb" containerName="console"
Sep 18 15:40:23.455413 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.455398    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5z52m"
Sep 18 15:40:23.458008 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.457986    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 18 15:40:23.458144 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.458020    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 18 15:40:23.459030 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.459013    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-ljnl9\""
Sep 18 15:40:23.465542 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.465521    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5z52m"]
Sep 18 15:40:23.474463 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.474442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7kgv\" (UniqueName: \"kubernetes.io/projected/cdfa44ff-0f89-471a-ab2a-ae558e53d736-kube-api-access-q7kgv\") pod \"cert-manager-cainjector-559cdd8c87-5z52m\" (UID: \"cdfa44ff-0f89-471a-ab2a-ae558e53d736\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5z52m"
Sep 18 15:40:23.474568 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.474514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdfa44ff-0f89-471a-ab2a-ae558e53d736-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5z52m\" (UID: \"cdfa44ff-0f89-471a-ab2a-ae558e53d736\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5z52m"
Sep 18 15:40:23.575303 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.575269    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7kgv\" (UniqueName: \"kubernetes.io/projected/cdfa44ff-0f89-471a-ab2a-ae558e53d736-kube-api-access-q7kgv\") pod \"cert-manager-cainjector-559cdd8c87-5z52m\" (UID: \"cdfa44ff-0f89-471a-ab2a-ae558e53d736\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5z52m"
Sep 18 15:40:23.575495 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.575332    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdfa44ff-0f89-471a-ab2a-ae558e53d736-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5z52m\" (UID: \"cdfa44ff-0f89-471a-ab2a-ae558e53d736\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5z52m"
Sep 18 15:40:23.584455 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.584424    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cdfa44ff-0f89-471a-ab2a-ae558e53d736-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5z52m\" (UID: \"cdfa44ff-0f89-471a-ab2a-ae558e53d736\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5z52m"
Sep 18 15:40:23.584585 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.584567    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7kgv\" (UniqueName: \"kubernetes.io/projected/cdfa44ff-0f89-471a-ab2a-ae558e53d736-kube-api-access-q7kgv\") pod \"cert-manager-cainjector-559cdd8c87-5z52m\" (UID: \"cdfa44ff-0f89-471a-ab2a-ae558e53d736\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5z52m"
Sep 18 15:40:23.782497 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:23.782415    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5z52m"
Sep 18 15:40:24.052291 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:24.052265    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5z52m"]
Sep 18 15:40:24.054271 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:40:24.054246    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcdfa44ff_0f89_471a_ab2a_ae558e53d736.slice/crio-e4795d5908f50935b8422412017212b65e5acae70267338ab49f444fc95603ae WatchSource:0}: Error finding container e4795d5908f50935b8422412017212b65e5acae70267338ab49f444fc95603ae: Status 404 returned error can't find the container with id e4795d5908f50935b8422412017212b65e5acae70267338ab49f444fc95603ae
Sep 18 15:40:24.056123 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:24.056106    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 15:40:24.977253 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:24.977219    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5z52m" event={"ID":"cdfa44ff-0f89-471a-ab2a-ae558e53d736","Type":"ContainerStarted","Data":"e4795d5908f50935b8422412017212b65e5acae70267338ab49f444fc95603ae"}
Sep 18 15:40:28.995049 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:28.995018    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5z52m" event={"ID":"cdfa44ff-0f89-471a-ab2a-ae558e53d736","Type":"ContainerStarted","Data":"23677d9b75ee6dd51c8bc88adf5e58a9eafffe0bb33ac52a291dbe19bf9fe82a"}
Sep 18 15:40:29.015121 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:29.015077    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5z52m" podStartSLOduration=2.12872565 podStartE2EDuration="6.015062044s" totalImagesPullingTime="3.886336394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:40:29.013993966 +0000 UTC m=+372.559284630" watchObservedRunningTime="2026-09-18 15:40:29.015062044 +0000 UTC m=+372.560352718"
Sep 18 15:40:45.446785 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.446746    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c6b6bf45b-7h252"]
Sep 18 15:40:45.450328 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.450301    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.465307 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.465283    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6b6bf45b-7h252"]
Sep 18 15:40:45.556283 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.556252    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66e3bffa-358c-4cb3-875d-c9ad76901558-service-ca\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.556445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.556288    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp7fz\" (UniqueName: \"kubernetes.io/projected/66e3bffa-358c-4cb3-875d-c9ad76901558-kube-api-access-cp7fz\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.556445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.556321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66e3bffa-358c-4cb3-875d-c9ad76901558-oauth-serving-cert\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.556445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.556409    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e3bffa-358c-4cb3-875d-c9ad76901558-trusted-ca-bundle\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.556445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.556439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66e3bffa-358c-4cb3-875d-c9ad76901558-console-serving-cert\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.556605 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.556503    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66e3bffa-358c-4cb3-875d-c9ad76901558-console-oauth-config\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.556605 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.556523    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66e3bffa-358c-4cb3-875d-c9ad76901558-console-config\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.657923 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.657564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e3bffa-358c-4cb3-875d-c9ad76901558-trusted-ca-bundle\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.657923 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.657617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66e3bffa-358c-4cb3-875d-c9ad76901558-console-serving-cert\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.657923 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.657701    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66e3bffa-358c-4cb3-875d-c9ad76901558-console-oauth-config\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.657923 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.657731    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66e3bffa-358c-4cb3-875d-c9ad76901558-console-config\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.657923 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.657778    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66e3bffa-358c-4cb3-875d-c9ad76901558-service-ca\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.657923 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.657807    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp7fz\" (UniqueName: \"kubernetes.io/projected/66e3bffa-358c-4cb3-875d-c9ad76901558-kube-api-access-cp7fz\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.657923 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.657837    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66e3bffa-358c-4cb3-875d-c9ad76901558-oauth-serving-cert\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.658714 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.658683    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e3bffa-358c-4cb3-875d-c9ad76901558-trusted-ca-bundle\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.659184 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.659154    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66e3bffa-358c-4cb3-875d-c9ad76901558-service-ca\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.659982 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.659955    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66e3bffa-358c-4cb3-875d-c9ad76901558-console-config\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.660489 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.660465    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66e3bffa-358c-4cb3-875d-c9ad76901558-oauth-serving-cert\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.661243 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.661218    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66e3bffa-358c-4cb3-875d-c9ad76901558-console-serving-cert\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.662492 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.662467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66e3bffa-358c-4cb3-875d-c9ad76901558-console-oauth-config\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.666470 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.666452    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp7fz\" (UniqueName: \"kubernetes.io/projected/66e3bffa-358c-4cb3-875d-c9ad76901558-kube-api-access-cp7fz\") pod \"console-7c6b6bf45b-7h252\" (UID: \"66e3bffa-358c-4cb3-875d-c9ad76901558\") " pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:45.761177 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:45.761096    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:46.036843 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:46.036819    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6b6bf45b-7h252"]
Sep 18 15:40:46.039575 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:40:46.039545    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e3bffa_358c_4cb3_875d_c9ad76901558.slice/crio-6af903b2dc4e6306b2610145b66ef3a21bacc6be0a4149054ca69c96d3b49fda WatchSource:0}: Error finding container 6af903b2dc4e6306b2610145b66ef3a21bacc6be0a4149054ca69c96d3b49fda: Status 404 returned error can't find the container with id 6af903b2dc4e6306b2610145b66ef3a21bacc6be0a4149054ca69c96d3b49fda
Sep 18 15:40:46.066712 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:46.066686    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6b6bf45b-7h252" event={"ID":"66e3bffa-358c-4cb3-875d-c9ad76901558","Type":"ContainerStarted","Data":"6af903b2dc4e6306b2610145b66ef3a21bacc6be0a4149054ca69c96d3b49fda"}
Sep 18 15:40:47.073524 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:47.073490    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6b6bf45b-7h252" event={"ID":"66e3bffa-358c-4cb3-875d-c9ad76901558","Type":"ContainerStarted","Data":"8c402c4093323d7cfb029f1e8323cd35fab2a52135b138eb17491dd9020efca7"}
Sep 18 15:40:47.095784 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:47.095738    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c6b6bf45b-7h252" podStartSLOduration=2.095722758 podStartE2EDuration="2.095722758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:40:47.094094622 +0000 UTC m=+390.639385295" watchObservedRunningTime="2026-09-18 15:40:47.095722758 +0000 UTC m=+390.641013431"
Sep 18 15:40:55.761897 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:55.761864    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:55.762353 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:55.761921    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:55.766579 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:55.766556    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:56.115535 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:56.115452    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c6b6bf45b-7h252"
Sep 18 15:40:56.175696 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:56.175648    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54dfdb646c-bd42n"]
Sep 18 15:40:57.424001 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.423970    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"]
Sep 18 15:40:57.428782 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.428763    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.434032 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.434011    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 15:40:57.435091 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.435075    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 15:40:57.437177 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.437146    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 15:40:57.437272 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.437214    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cqmrf\""
Sep 18 15:40:57.437329 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.437276    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 15:40:57.449768 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.449739    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"]
Sep 18 15:40:57.563868 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.563833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e1bdfc0-5b16-44cd-8469-05647e693619-manifests\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.564046 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.563899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e1bdfc0-5b16-44cd-8469-05647e693619-tmp\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.564046 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.563922    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e1bdfc0-5b16-44cd-8469-05647e693619-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.564046 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.563941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e1bdfc0-5b16-44cd-8469-05647e693619-webhook-cert\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.564046 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.563962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k89cp\" (UniqueName: \"kubernetes.io/projected/9e1bdfc0-5b16-44cd-8469-05647e693619-kube-api-access-k89cp\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.664626 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.664591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e1bdfc0-5b16-44cd-8469-05647e693619-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.664626 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.664627    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e1bdfc0-5b16-44cd-8469-05647e693619-webhook-cert\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.664876 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.664647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k89cp\" (UniqueName: \"kubernetes.io/projected/9e1bdfc0-5b16-44cd-8469-05647e693619-kube-api-access-k89cp\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.664876 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.664718    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e1bdfc0-5b16-44cd-8469-05647e693619-manifests\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.664876 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.664766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e1bdfc0-5b16-44cd-8469-05647e693619-tmp\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.665184 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.665156    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e1bdfc0-5b16-44cd-8469-05647e693619-manifests\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.666916 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.666896    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e1bdfc0-5b16-44cd-8469-05647e693619-tmp\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.667229 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.667213    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e1bdfc0-5b16-44cd-8469-05647e693619-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.667271 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.667240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e1bdfc0-5b16-44cd-8469-05647e693619-webhook-cert\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.672746 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.672722    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k89cp\" (UniqueName: \"kubernetes.io/projected/9e1bdfc0-5b16-44cd-8469-05647e693619-kube-api-access-k89cp\") pod \"opendatahub-operator-controller-manager-54b8c8d665-mdph5\" (UID: \"9e1bdfc0-5b16-44cd-8469-05647e693619\") " pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:57.739135 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:57.739052    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:40:58.016391 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:58.016364    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"]
Sep 18 15:40:58.019569 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:40:58.019541    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1bdfc0_5b16_44cd_8469_05647e693619.slice/crio-98270abf1b61160472138ba88f3bf2850632f39bcd408a8edcfc4742c72a70b4 WatchSource:0}: Error finding container 98270abf1b61160472138ba88f3bf2850632f39bcd408a8edcfc4742c72a70b4: Status 404 returned error can't find the container with id 98270abf1b61160472138ba88f3bf2850632f39bcd408a8edcfc4742c72a70b4
Sep 18 15:40:58.121210 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:40:58.121173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5" event={"ID":"9e1bdfc0-5b16-44cd-8469-05647e693619","Type":"ContainerStarted","Data":"98270abf1b61160472138ba88f3bf2850632f39bcd408a8edcfc4742c72a70b4"}
Sep 18 15:41:02.143601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:02.143567    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5" event={"ID":"9e1bdfc0-5b16-44cd-8469-05647e693619","Type":"ContainerStarted","Data":"82abe1f2bb4ea39b502c3e953f4dda69f59317b5026e802645ff2949febee64c"}
Sep 18 15:41:03.152700 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:03.152649    2561 generic.go:356] "Generic (PLEG): container finished" podID="9e1bdfc0-5b16-44cd-8469-05647e693619" containerID="82abe1f2bb4ea39b502c3e953f4dda69f59317b5026e802645ff2949febee64c" exitCode=0
Sep 18 15:41:03.153103 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:03.152735    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5" event={"ID":"9e1bdfc0-5b16-44cd-8469-05647e693619","Type":"ContainerDied","Data":"82abe1f2bb4ea39b502c3e953f4dda69f59317b5026e802645ff2949febee64c"}
Sep 18 15:41:04.158927 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:04.158889    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5" event={"ID":"9e1bdfc0-5b16-44cd-8469-05647e693619","Type":"ContainerStarted","Data":"bda70e160c8c9c10e098b22b40060371cfeb84b250e2f564ca33828adc46db21"}
Sep 18 15:41:04.159392 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:04.159002    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:41:04.189537 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:04.189491    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5" podStartSLOduration=2.428069832 podStartE2EDuration="7.189475923s" totalImagesPullingTime="3.756922912s" totalInitContainerRuntime="1.004483179s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:40:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:41:04.188704523 +0000 UTC m=+407.733995197" watchObservedRunningTime="2026-09-18 15:41:04.189475923 +0000 UTC m=+407.734766656"
Sep 18 15:41:15.166069 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:15.166042    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54b8c8d665-mdph5"
Sep 18 15:41:17.685894 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.685856    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"]
Sep 18 15:41:17.693151 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.693121    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"
Sep 18 15:41:17.695810 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.695788    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 15:41:17.695935 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.695790    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 15:41:17.706444 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.706421    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"]
Sep 18 15:41:17.740487 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.740457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cdb2\" (UniqueName: \"kubernetes.io/projected/7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29-kube-api-access-4cdb2\") pod \"kube-auth-proxy-7fbb59494c-kv8cx\" (UID: \"7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29\") " pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"
Sep 18 15:41:17.740718 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.740536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29-tls-certs\") pod \"kube-auth-proxy-7fbb59494c-kv8cx\" (UID: \"7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29\") " pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"
Sep 18 15:41:17.740718 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.740604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29-tmp\") pod \"kube-auth-proxy-7fbb59494c-kv8cx\" (UID: \"7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29\") " pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"
Sep 18 15:41:17.841383 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.841346    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29-tmp\") pod \"kube-auth-proxy-7fbb59494c-kv8cx\" (UID: \"7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29\") " pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"
Sep 18 15:41:17.841555 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.841417    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cdb2\" (UniqueName: \"kubernetes.io/projected/7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29-kube-api-access-4cdb2\") pod \"kube-auth-proxy-7fbb59494c-kv8cx\" (UID: \"7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29\") " pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"
Sep 18 15:41:17.841555 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.841446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29-tls-certs\") pod \"kube-auth-proxy-7fbb59494c-kv8cx\" (UID: \"7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29\") " pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"
Sep 18 15:41:17.843642 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.843615    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29-tmp\") pod \"kube-auth-proxy-7fbb59494c-kv8cx\" (UID: \"7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29\") " pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"
Sep 18 15:41:17.843970 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.843949    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29-tls-certs\") pod \"kube-auth-proxy-7fbb59494c-kv8cx\" (UID: \"7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29\") " pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"
Sep 18 15:41:17.860479 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:17.860456    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cdb2\" (UniqueName: \"kubernetes.io/projected/7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29-kube-api-access-4cdb2\") pod \"kube-auth-proxy-7fbb59494c-kv8cx\" (UID: \"7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29\") " pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"
Sep 18 15:41:18.002348 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:18.002267    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"
Sep 18 15:41:18.291301 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:18.291217    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx"]
Sep 18 15:41:18.297333 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:41:18.297307    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5e0c89_88c9_4f9e_a4b3_6ea9c1f46f29.slice/crio-da4a7a12447358d2bbeb98b20ab4d853ed1bd0db8f1546f4edbbd8b9b3563155 WatchSource:0}: Error finding container da4a7a12447358d2bbeb98b20ab4d853ed1bd0db8f1546f4edbbd8b9b3563155: Status 404 returned error can't find the container with id da4a7a12447358d2bbeb98b20ab4d853ed1bd0db8f1546f4edbbd8b9b3563155
Sep 18 15:41:19.226559 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.226519    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx" event={"ID":"7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29","Type":"ContainerStarted","Data":"da4a7a12447358d2bbeb98b20ab4d853ed1bd0db8f1546f4edbbd8b9b3563155"}
Sep 18 15:41:19.383599 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.383569    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"]
Sep 18 15:41:19.387704 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.387679    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.392577 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.392552    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 15:41:19.392809 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.392792    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 15:41:19.393389 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.392974    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 15:41:19.393389 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.393246    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8s7zz\""
Sep 18 15:41:19.393813 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.393796    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 15:41:19.405969 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.405867    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"]
Sep 18 15:41:19.457040 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.456986    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxvlf\" (UniqueName: \"kubernetes.io/projected/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-kube-api-access-lxvlf\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.457202 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.457055    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.457202 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.457104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.457202 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.457164    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.457388 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.457223    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.457388 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.457304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.457388 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.457372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.558443 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.558367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxvlf\" (UniqueName: \"kubernetes.io/projected/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-kube-api-access-lxvlf\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.558443 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.558424    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.558678 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.558471    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.558678 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.558492    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.558678 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.558517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.558678 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.558541    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.558678 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.558564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.559604 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.559533    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.561318 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.561274    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.561490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.561465    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.561542 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.561527    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.561628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.561610    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.567173 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.567143    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.567496 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.567471    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxvlf\" (UniqueName: \"kubernetes.io/projected/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-kube-api-access-lxvlf\") pod \"istiod-openshift-gateway-7b7995d5cd-wrzbn\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:19.702293 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:19.702247    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:20.005372 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:20.005346    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"]
Sep 18 15:41:21.195000 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:41:21.194971    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbd3cdf_5c65_4696_ad60_fc3aa5841e5a.slice/crio-c5cda7f594d2275bf4699f22f94e80cf5ae79c50f88b04564112146b2c977cf2 WatchSource:0}: Error finding container c5cda7f594d2275bf4699f22f94e80cf5ae79c50f88b04564112146b2c977cf2: Status 404 returned error can't find the container with id c5cda7f594d2275bf4699f22f94e80cf5ae79c50f88b04564112146b2c977cf2
Sep 18 15:41:21.204537 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.204482    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54dfdb646c-bd42n" podUID="b71954b1-f3c0-451a-a734-b6f5c392a8cf" containerName="console" containerID="cri-o://4bc43c7f6949cf03758f3b8d4c0ed3ec1704863759ba24b1cdf6b13fb00fe2f6" gracePeriod=15
Sep 18 15:41:21.238066 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.238033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn" event={"ID":"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a","Type":"ContainerStarted","Data":"c5cda7f594d2275bf4699f22f94e80cf5ae79c50f88b04564112146b2c977cf2"}
Sep 18 15:41:21.251517 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.251496    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 15:41:21.460737 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.460713    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:41:21.577956 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.577922    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-oauth-serving-cert\") pod \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") "
Sep 18 15:41:21.577956 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.577959    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-trusted-ca-bundle\") pod \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") "
Sep 18 15:41:21.578152 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.578008    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-config\") pod \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") "
Sep 18 15:41:21.578152 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.578024    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22tng\" (UniqueName: \"kubernetes.io/projected/b71954b1-f3c0-451a-a734-b6f5c392a8cf-kube-api-access-22tng\") pod \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") "
Sep 18 15:41:21.578152 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.578051    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-service-ca\") pod \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") "
Sep 18 15:41:21.578152 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.578079    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-serving-cert\") pod \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") "
Sep 18 15:41:21.578152 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.578151    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-oauth-config\") pod \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\" (UID: \"b71954b1-f3c0-451a-a734-b6f5c392a8cf\") "
Sep 18 15:41:21.578502 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.578473    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b71954b1-f3c0-451a-a734-b6f5c392a8cf" (UID: "b71954b1-f3c0-451a-a734-b6f5c392a8cf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:41:21.578583 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.578552    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-config" (OuterVolumeSpecName: "console-config") pod "b71954b1-f3c0-451a-a734-b6f5c392a8cf" (UID: "b71954b1-f3c0-451a-a734-b6f5c392a8cf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:41:21.578640 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.578620    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b71954b1-f3c0-451a-a734-b6f5c392a8cf" (UID: "b71954b1-f3c0-451a-a734-b6f5c392a8cf"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:41:21.578716 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.578638    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-service-ca" (OuterVolumeSpecName: "service-ca") pod "b71954b1-f3c0-451a-a734-b6f5c392a8cf" (UID: "b71954b1-f3c0-451a-a734-b6f5c392a8cf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:41:21.580295 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.580275    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b71954b1-f3c0-451a-a734-b6f5c392a8cf" (UID: "b71954b1-f3c0-451a-a734-b6f5c392a8cf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:41:21.580698 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.580680    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b71954b1-f3c0-451a-a734-b6f5c392a8cf" (UID: "b71954b1-f3c0-451a-a734-b6f5c392a8cf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:41:21.580698 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.580684    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71954b1-f3c0-451a-a734-b6f5c392a8cf-kube-api-access-22tng" (OuterVolumeSpecName: "kube-api-access-22tng") pod "b71954b1-f3c0-451a-a734-b6f5c392a8cf" (UID: "b71954b1-f3c0-451a-a734-b6f5c392a8cf"). InnerVolumeSpecName "kube-api-access-22tng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:41:21.679432 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.679405    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-oauth-config\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:21.679432 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.679432    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-oauth-serving-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:21.679628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.679446    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-trusted-ca-bundle\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:21.679628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.679457    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-config\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:21.679628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.679469    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22tng\" (UniqueName: \"kubernetes.io/projected/b71954b1-f3c0-451a-a734-b6f5c392a8cf-kube-api-access-22tng\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:21.679628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.679480    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b71954b1-f3c0-451a-a734-b6f5c392a8cf-service-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:21.679628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:21.679491    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b71954b1-f3c0-451a-a734-b6f5c392a8cf-console-serving-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:22.245002 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:22.244965    2561 generic.go:356] "Generic (PLEG): container finished" podID="b71954b1-f3c0-451a-a734-b6f5c392a8cf" containerID="4bc43c7f6949cf03758f3b8d4c0ed3ec1704863759ba24b1cdf6b13fb00fe2f6" exitCode=0
Sep 18 15:41:22.245440 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:22.245018    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54dfdb646c-bd42n"
Sep 18 15:41:22.245440 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:22.245079    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dfdb646c-bd42n" event={"ID":"b71954b1-f3c0-451a-a734-b6f5c392a8cf","Type":"ContainerDied","Data":"4bc43c7f6949cf03758f3b8d4c0ed3ec1704863759ba24b1cdf6b13fb00fe2f6"}
Sep 18 15:41:22.245440 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:22.245125    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54dfdb646c-bd42n" event={"ID":"b71954b1-f3c0-451a-a734-b6f5c392a8cf","Type":"ContainerDied","Data":"90290fe70338c34323d475d748e57849328b229230a42f8e834c7f13c4a8ce9d"}
Sep 18 15:41:22.245440 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:22.245140    2561 scope.go:117] "RemoveContainer" containerID="4bc43c7f6949cf03758f3b8d4c0ed3ec1704863759ba24b1cdf6b13fb00fe2f6"
Sep 18 15:41:22.247138 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:22.247114    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx" event={"ID":"7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29","Type":"ContainerStarted","Data":"dafbb391bb8d78e7b7c84077938b1d9e1b335ce4e7e8a9c8dd651eef8973d869"}
Sep 18 15:41:22.269744 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:22.269688    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7fbb59494c-kv8cx" podStartSLOduration=2.32010354 podStartE2EDuration="5.269641931s" totalImagesPullingTime="2.949538391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:41:22.268753227 +0000 UTC m=+425.814043902" watchObservedRunningTime="2026-09-18 15:41:22.269641931 +0000 UTC m=+425.814932630"
Sep 18 15:41:22.273480 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:22.273458    2561 scope.go:117] "RemoveContainer" containerID="4bc43c7f6949cf03758f3b8d4c0ed3ec1704863759ba24b1cdf6b13fb00fe2f6"
Sep 18 15:41:22.273875 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:41:22.273848    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc43c7f6949cf03758f3b8d4c0ed3ec1704863759ba24b1cdf6b13fb00fe2f6\": container with ID starting with 4bc43c7f6949cf03758f3b8d4c0ed3ec1704863759ba24b1cdf6b13fb00fe2f6 not found: ID does not exist" containerID="4bc43c7f6949cf03758f3b8d4c0ed3ec1704863759ba24b1cdf6b13fb00fe2f6"
Sep 18 15:41:22.273963 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:22.273878    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc43c7f6949cf03758f3b8d4c0ed3ec1704863759ba24b1cdf6b13fb00fe2f6"} err="failed to get container status \"4bc43c7f6949cf03758f3b8d4c0ed3ec1704863759ba24b1cdf6b13fb00fe2f6\": rpc error: code = NotFound desc = could not find container \"4bc43c7f6949cf03758f3b8d4c0ed3ec1704863759ba24b1cdf6b13fb00fe2f6\": container with ID starting with 4bc43c7f6949cf03758f3b8d4c0ed3ec1704863759ba24b1cdf6b13fb00fe2f6 not found: ID does not exist"
Sep 18 15:41:22.287278 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:22.287250    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54dfdb646c-bd42n"]
Sep 18 15:41:22.302797 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:22.300237    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54dfdb646c-bd42n"]
Sep 18 15:41:22.380826 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:22.380792    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b71954b1-f3c0-451a-a734-b6f5c392a8cf" path="/var/lib/kubelet/pods/b71954b1-f3c0-451a-a734-b6f5c392a8cf/volumes"
Sep 18 15:41:23.680547 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:23.680511    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:41:23.680921 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:23.680591    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:41:24.259165 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:24.259126    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn" event={"ID":"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a","Type":"ContainerStarted","Data":"36c02fc086277176c413d7ac0d1a40ec9459c13b8f4222df47f8c3802504536d"}
Sep 18 15:41:24.259334 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:24.259183    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:24.261088 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:24.261058    2561 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-wrzbn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 15:41:24.261190 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:24.261119    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn" podUID="5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 15:41:24.282584 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:24.282541    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn" podStartSLOduration=2.799327451 podStartE2EDuration="5.282526993s" totalImagesPullingTime="2.483199542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:41:24.281148921 +0000 UTC m=+427.826439596" watchObservedRunningTime="2026-09-18 15:41:24.282526993 +0000 UTC m=+427.827817661"
Sep 18 15:41:25.264787 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:25.264763    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:36.199811 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.199733    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"]
Sep 18 15:41:36.200257 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.200118    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71954b1-f3c0-451a-a734-b6f5c392a8cf" containerName="console"
Sep 18 15:41:36.200257 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.200128    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71954b1-f3c0-451a-a734-b6f5c392a8cf" containerName="console"
Sep 18 15:41:36.200257 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.200180    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71954b1-f3c0-451a-a734-b6f5c392a8cf" containerName="console"
Sep 18 15:41:36.203048 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.203031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.207567 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.207545    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jspcm\""
Sep 18 15:41:36.208089 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.208073    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 15:41:36.208450 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.208428    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 15:41:36.208605 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.208482    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 15:41:36.208605 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.208428    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 15:41:36.208776 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.208743    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 15:41:36.220066 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.220038    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"]
Sep 18 15:41:36.309122 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.309095    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15d48601-1baf-4949-98d5-d85083be374d-cert\") pod \"lws-controller-manager-c59846bb8-dzjgk\" (UID: \"15d48601-1baf-4949-98d5-d85083be374d\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.309291 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.309136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/15d48601-1baf-4949-98d5-d85083be374d-metrics-cert\") pod \"lws-controller-manager-c59846bb8-dzjgk\" (UID: \"15d48601-1baf-4949-98d5-d85083be374d\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.309291 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.309174    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2nfx\" (UniqueName: \"kubernetes.io/projected/15d48601-1baf-4949-98d5-d85083be374d-kube-api-access-x2nfx\") pod \"lws-controller-manager-c59846bb8-dzjgk\" (UID: \"15d48601-1baf-4949-98d5-d85083be374d\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.309291 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.309243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/15d48601-1baf-4949-98d5-d85083be374d-manager-config\") pod \"lws-controller-manager-c59846bb8-dzjgk\" (UID: \"15d48601-1baf-4949-98d5-d85083be374d\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.410551 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.410518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2nfx\" (UniqueName: \"kubernetes.io/projected/15d48601-1baf-4949-98d5-d85083be374d-kube-api-access-x2nfx\") pod \"lws-controller-manager-c59846bb8-dzjgk\" (UID: \"15d48601-1baf-4949-98d5-d85083be374d\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.410748 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.410572    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/15d48601-1baf-4949-98d5-d85083be374d-manager-config\") pod \"lws-controller-manager-c59846bb8-dzjgk\" (UID: \"15d48601-1baf-4949-98d5-d85083be374d\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.410748 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.410612    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15d48601-1baf-4949-98d5-d85083be374d-cert\") pod \"lws-controller-manager-c59846bb8-dzjgk\" (UID: \"15d48601-1baf-4949-98d5-d85083be374d\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.410748 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.410635    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/15d48601-1baf-4949-98d5-d85083be374d-metrics-cert\") pod \"lws-controller-manager-c59846bb8-dzjgk\" (UID: \"15d48601-1baf-4949-98d5-d85083be374d\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.411333 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.411303    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/15d48601-1baf-4949-98d5-d85083be374d-manager-config\") pod \"lws-controller-manager-c59846bb8-dzjgk\" (UID: \"15d48601-1baf-4949-98d5-d85083be374d\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.413197 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.413177    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15d48601-1baf-4949-98d5-d85083be374d-cert\") pod \"lws-controller-manager-c59846bb8-dzjgk\" (UID: \"15d48601-1baf-4949-98d5-d85083be374d\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.413285 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.413257    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/15d48601-1baf-4949-98d5-d85083be374d-metrics-cert\") pod \"lws-controller-manager-c59846bb8-dzjgk\" (UID: \"15d48601-1baf-4949-98d5-d85083be374d\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.424466 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.424442    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2nfx\" (UniqueName: \"kubernetes.io/projected/15d48601-1baf-4949-98d5-d85083be374d-kube-api-access-x2nfx\") pod \"lws-controller-manager-c59846bb8-dzjgk\" (UID: \"15d48601-1baf-4949-98d5-d85083be374d\") " pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.512206 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.512132    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:36.785312 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:36.785291    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"]
Sep 18 15:41:36.787464 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:41:36.787442    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d48601_1baf_4949_98d5_d85083be374d.slice/crio-73dd00d52849eacf0fc5fb3526c39a8d50f07ed13151bf27a4d121efee39c5c4 WatchSource:0}: Error finding container 73dd00d52849eacf0fc5fb3526c39a8d50f07ed13151bf27a4d121efee39c5c4: Status 404 returned error can't find the container with id 73dd00d52849eacf0fc5fb3526c39a8d50f07ed13151bf27a4d121efee39c5c4
Sep 18 15:41:37.318414 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:37.318375    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk" event={"ID":"15d48601-1baf-4949-98d5-d85083be374d","Type":"ContainerStarted","Data":"73dd00d52849eacf0fc5fb3526c39a8d50f07ed13151bf27a4d121efee39c5c4"}
Sep 18 15:41:39.330740 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.330626    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk" event={"ID":"15d48601-1baf-4949-98d5-d85083be374d","Type":"ContainerStarted","Data":"93434303eb6cf6c30b455017932a310373e60559ce8d598a33058ff4e49b368b"}
Sep 18 15:41:39.331086 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.330753    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:41:39.363789 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.363723    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk" podStartSLOduration=1.075069049 podStartE2EDuration="3.363703454s" totalImagesPullingTime="2.288634405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:41:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:41:39.363397173 +0000 UTC m=+442.908687853" watchObservedRunningTime="2026-09-18 15:41:39.363703454 +0000 UTC m=+442.908994128"
Sep 18 15:41:39.394141 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.394106    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"]
Sep 18 15:41:39.394328 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.394306    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn" podUID="5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" containerName="discovery" containerID="cri-o://36c02fc086277176c413d7ac0d1a40ec9459c13b8f4222df47f8c3802504536d" gracePeriod=30
Sep 18 15:41:39.633919 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.633894    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:39.739474 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.739443    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxvlf\" (UniqueName: \"kubernetes.io/projected/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-kube-api-access-lxvlf\") pod \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") "
Sep 18 15:41:39.739474 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.739475    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-local-certs\") pod \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") "
Sep 18 15:41:39.739754 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.739521    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-csr-dns-cert\") pod \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") "
Sep 18 15:41:39.739754 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.739547    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-token\") pod \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") "
Sep 18 15:41:39.739754 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.739564    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-csr-ca-configmap\") pod \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") "
Sep 18 15:41:39.739754 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.739584    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-kubeconfig\") pod \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") "
Sep 18 15:41:39.739754 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.739632    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-cacerts\") pod \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\" (UID: \"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a\") "
Sep 18 15:41:39.740105 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.740070    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" (UID: "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:41:39.741990 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.741962    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-cacerts" (OuterVolumeSpecName: "cacerts") pod "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" (UID: "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:41:39.742098 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.741982    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-local-certs" (OuterVolumeSpecName: "local-certs") pod "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" (UID: "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:41:39.742153 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.742131    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" (UID: "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:41:39.742208 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.742150    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-kube-api-access-lxvlf" (OuterVolumeSpecName: "kube-api-access-lxvlf") pod "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" (UID: "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a"). InnerVolumeSpecName "kube-api-access-lxvlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:41:39.742246 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.742221    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" (UID: "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:41:39.742246 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.742234    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" (UID: "5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:41:39.840532 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.840496    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-csr-dns-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:39.840532 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.840526    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-token\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:39.840532 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.840536    2561 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-csr-ca-configmap\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:39.840775 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.840545    2561 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-istio-kubeconfig\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:39.840775 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.840553    2561 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-cacerts\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:39.840775 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.840560    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxvlf\" (UniqueName: \"kubernetes.io/projected/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-kube-api-access-lxvlf\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:39.840775 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:39.840570    2561 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a-local-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:41:40.336623 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:40.336528    2561 generic.go:356] "Generic (PLEG): container finished" podID="5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" containerID="36c02fc086277176c413d7ac0d1a40ec9459c13b8f4222df47f8c3802504536d" exitCode=0
Sep 18 15:41:40.336623 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:40.336584    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"
Sep 18 15:41:40.337149 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:40.336625    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn" event={"ID":"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a","Type":"ContainerDied","Data":"36c02fc086277176c413d7ac0d1a40ec9459c13b8f4222df47f8c3802504536d"}
Sep 18 15:41:40.337149 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:40.336690    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn" event={"ID":"5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a","Type":"ContainerDied","Data":"c5cda7f594d2275bf4699f22f94e80cf5ae79c50f88b04564112146b2c977cf2"}
Sep 18 15:41:40.337149 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:40.336716    2561 scope.go:117] "RemoveContainer" containerID="36c02fc086277176c413d7ac0d1a40ec9459c13b8f4222df47f8c3802504536d"
Sep 18 15:41:40.346264 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:40.346242    2561 scope.go:117] "RemoveContainer" containerID="36c02fc086277176c413d7ac0d1a40ec9459c13b8f4222df47f8c3802504536d"
Sep 18 15:41:40.349010 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:41:40.348980    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c02fc086277176c413d7ac0d1a40ec9459c13b8f4222df47f8c3802504536d\": container with ID starting with 36c02fc086277176c413d7ac0d1a40ec9459c13b8f4222df47f8c3802504536d not found: ID does not exist" containerID="36c02fc086277176c413d7ac0d1a40ec9459c13b8f4222df47f8c3802504536d"
Sep 18 15:41:40.349122 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:40.349020    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c02fc086277176c413d7ac0d1a40ec9459c13b8f4222df47f8c3802504536d"} err="failed to get container status \"36c02fc086277176c413d7ac0d1a40ec9459c13b8f4222df47f8c3802504536d\": rpc error: code = NotFound desc = could not find container \"36c02fc086277176c413d7ac0d1a40ec9459c13b8f4222df47f8c3802504536d\": container with ID starting with 36c02fc086277176c413d7ac0d1a40ec9459c13b8f4222df47f8c3802504536d not found: ID does not exist"
Sep 18 15:41:40.360391 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:40.360368    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"]
Sep 18 15:41:40.366445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:40.366425    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-wrzbn"]
Sep 18 15:41:40.379103 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:40.379079    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" path="/var/lib/kubelet/pods/5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a/volumes"
Sep 18 15:41:50.339195 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:41:50.339167    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-c59846bb8-dzjgk"
Sep 18 15:42:58.291647 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.291614    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"]
Sep 18 15:42:58.292601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.292572    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" containerName="discovery"
Sep 18 15:42:58.292601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.292602    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" containerName="discovery"
Sep 18 15:42:58.292854 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.292840    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bbd3cdf-5c65-4696-ad60-fc3aa5841e5a" containerName="discovery"
Sep 18 15:42:58.296653 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.296625    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"
Sep 18 15:42:58.299786 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.299758    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 15:42:58.299903 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.299758    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 18 15:42:58.299903 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.299765    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 18 15:42:58.300817 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.300801    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f6brr\""
Sep 18 15:42:58.300886 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.300802    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 15:42:58.306327 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.306307    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"]
Sep 18 15:42:58.389221 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.389194    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5plkr\" (UID: \"7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"
Sep 18 15:42:58.389413 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.389238    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5plkr\" (UID: \"7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"
Sep 18 15:42:58.389413 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.389264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwjcb\" (UniqueName: \"kubernetes.io/projected/7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc-kube-api-access-nwjcb\") pod \"kuadrant-console-plugin-fbdd4f445-5plkr\" (UID: \"7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"
Sep 18 15:42:58.490111 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.490081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5plkr\" (UID: \"7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"
Sep 18 15:42:58.490273 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.490132    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5plkr\" (UID: \"7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"
Sep 18 15:42:58.490273 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.490176    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwjcb\" (UniqueName: \"kubernetes.io/projected/7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc-kube-api-access-nwjcb\") pod \"kuadrant-console-plugin-fbdd4f445-5plkr\" (UID: \"7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"
Sep 18 15:42:58.490878 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.490857    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-5plkr\" (UID: \"7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"
Sep 18 15:42:58.492627 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.492607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-5plkr\" (UID: \"7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"
Sep 18 15:42:58.499190 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.499166    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwjcb\" (UniqueName: \"kubernetes.io/projected/7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc-kube-api-access-nwjcb\") pod \"kuadrant-console-plugin-fbdd4f445-5plkr\" (UID: \"7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"
Sep 18 15:42:58.607356 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.607278    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"
Sep 18 15:42:58.882206 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:58.882176    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr"]
Sep 18 15:42:58.883894 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:42:58.883867    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7785d30c_b3a7_4c18_b334_1e6fd1f9bcdc.slice/crio-371f5608856da57680ac88d2ca64d47ab93ec8fb85b3b2304492ab5c6be0d381 WatchSource:0}: Error finding container 371f5608856da57680ac88d2ca64d47ab93ec8fb85b3b2304492ab5c6be0d381: Status 404 returned error can't find the container with id 371f5608856da57680ac88d2ca64d47ab93ec8fb85b3b2304492ab5c6be0d381
Sep 18 15:42:59.699478 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:42:59.699437    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr" event={"ID":"7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc","Type":"ContainerStarted","Data":"371f5608856da57680ac88d2ca64d47ab93ec8fb85b3b2304492ab5c6be0d381"}
Sep 18 15:43:13.769490 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:13.769454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr" event={"ID":"7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc","Type":"ContainerStarted","Data":"b2feb068fbaed97b9560a19267e47a5934eda286a12f119b87d02e80fac25e56"}
Sep 18 15:43:13.789886 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:13.789832    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-5plkr" podStartSLOduration=1.302460501 podStartE2EDuration="15.78981599s" totalImagesPullingTime="14.487355489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:43:13.788290104 +0000 UTC m=+537.333580774" watchObservedRunningTime="2026-09-18 15:43:13.78981599 +0000 UTC m=+537.335106663"
Sep 18 15:43:57.870899 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:57.870866    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9cslp"]
Sep 18 15:43:57.888682 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:57.888646    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9cslp"]
Sep 18 15:43:57.888820 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:57.888756    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-9cslp"
Sep 18 15:43:57.891329 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:57.891305    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 15:43:57.967723 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:57.967691    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9cslp"]
Sep 18 15:43:57.998079 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:57.998055    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1943f126-8a84-429d-b8f0-270b9daeaf7e-config-file\") pod \"limitador-limitador-8588749977-9cslp\" (UID: \"1943f126-8a84-429d-b8f0-270b9daeaf7e\") " pod="kuadrant-system/limitador-limitador-8588749977-9cslp"
Sep 18 15:43:57.998232 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:57.998138    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z9fb\" (UniqueName: \"kubernetes.io/projected/1943f126-8a84-429d-b8f0-270b9daeaf7e-kube-api-access-2z9fb\") pod \"limitador-limitador-8588749977-9cslp\" (UID: \"1943f126-8a84-429d-b8f0-270b9daeaf7e\") " pod="kuadrant-system/limitador-limitador-8588749977-9cslp"
Sep 18 15:43:58.098880 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.098851    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z9fb\" (UniqueName: \"kubernetes.io/projected/1943f126-8a84-429d-b8f0-270b9daeaf7e-kube-api-access-2z9fb\") pod \"limitador-limitador-8588749977-9cslp\" (UID: \"1943f126-8a84-429d-b8f0-270b9daeaf7e\") " pod="kuadrant-system/limitador-limitador-8588749977-9cslp"
Sep 18 15:43:58.099034 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.098887    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1943f126-8a84-429d-b8f0-270b9daeaf7e-config-file\") pod \"limitador-limitador-8588749977-9cslp\" (UID: \"1943f126-8a84-429d-b8f0-270b9daeaf7e\") " pod="kuadrant-system/limitador-limitador-8588749977-9cslp"
Sep 18 15:43:58.099552 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.099534    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1943f126-8a84-429d-b8f0-270b9daeaf7e-config-file\") pod \"limitador-limitador-8588749977-9cslp\" (UID: \"1943f126-8a84-429d-b8f0-270b9daeaf7e\") " pod="kuadrant-system/limitador-limitador-8588749977-9cslp"
Sep 18 15:43:58.107242 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.107219    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z9fb\" (UniqueName: \"kubernetes.io/projected/1943f126-8a84-429d-b8f0-270b9daeaf7e-kube-api-access-2z9fb\") pod \"limitador-limitador-8588749977-9cslp\" (UID: \"1943f126-8a84-429d-b8f0-270b9daeaf7e\") " pod="kuadrant-system/limitador-limitador-8588749977-9cslp"
Sep 18 15:43:58.201356 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.201333    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-9cslp"
Sep 18 15:43:58.471373 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.471348    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9cslp"]
Sep 18 15:43:58.474408 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:43:58.474382    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1943f126_8a84_429d_b8f0_270b9daeaf7e.slice/crio-ed226e5cc59b9855d91bd0a4d9d01621513d953ecccd2d2bedbea213b3aa547e WatchSource:0}: Error finding container ed226e5cc59b9855d91bd0a4d9d01621513d953ecccd2d2bedbea213b3aa547e: Status 404 returned error can't find the container with id ed226e5cc59b9855d91bd0a4d9d01621513d953ecccd2d2bedbea213b3aa547e
Sep 18 15:43:58.671444 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.671413    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-875ng"]
Sep 18 15:43:58.675829 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.675813    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-875ng"
Sep 18 15:43:58.678288 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.678270    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-md7td\""
Sep 18 15:43:58.683100 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.683076    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-875ng"]
Sep 18 15:43:58.795013 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.794942    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-2jncm"]
Sep 18 15:43:58.798275 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.798260    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2jncm"
Sep 18 15:43:58.804740 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.804136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wsjb\" (UniqueName: \"kubernetes.io/projected/ecd4987a-3d8f-4ac6-9898-c9eb61be88fe-kube-api-access-6wsjb\") pod \"authorino-7bcfd8f9cd-875ng\" (UID: \"ecd4987a-3d8f-4ac6-9898-c9eb61be88fe\") " pod="kuadrant-system/authorino-7bcfd8f9cd-875ng"
Sep 18 15:43:58.806763 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.806739    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2jncm"]
Sep 18 15:43:58.905732 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.905705    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwghb\" (UniqueName: \"kubernetes.io/projected/8038e00b-21c6-4743-9109-f7a60d1d5e32-kube-api-access-cwghb\") pod \"authorino-59c974bd48-2jncm\" (UID: \"8038e00b-21c6-4743-9109-f7a60d1d5e32\") " pod="kuadrant-system/authorino-59c974bd48-2jncm"
Sep 18 15:43:58.906120 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.905747    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wsjb\" (UniqueName: \"kubernetes.io/projected/ecd4987a-3d8f-4ac6-9898-c9eb61be88fe-kube-api-access-6wsjb\") pod \"authorino-7bcfd8f9cd-875ng\" (UID: \"ecd4987a-3d8f-4ac6-9898-c9eb61be88fe\") " pod="kuadrant-system/authorino-7bcfd8f9cd-875ng"
Sep 18 15:43:58.914848 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.914823    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wsjb\" (UniqueName: \"kubernetes.io/projected/ecd4987a-3d8f-4ac6-9898-c9eb61be88fe-kube-api-access-6wsjb\") pod \"authorino-7bcfd8f9cd-875ng\" (UID: \"ecd4987a-3d8f-4ac6-9898-c9eb61be88fe\") " pod="kuadrant-system/authorino-7bcfd8f9cd-875ng"
Sep 18 15:43:58.972744 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.972713    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-9cslp" event={"ID":"1943f126-8a84-429d-b8f0-270b9daeaf7e","Type":"ContainerStarted","Data":"ed226e5cc59b9855d91bd0a4d9d01621513d953ecccd2d2bedbea213b3aa547e"}
Sep 18 15:43:58.986945 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:58.986923    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-875ng"
Sep 18 15:43:59.007011 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:59.006989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwghb\" (UniqueName: \"kubernetes.io/projected/8038e00b-21c6-4743-9109-f7a60d1d5e32-kube-api-access-cwghb\") pod \"authorino-59c974bd48-2jncm\" (UID: \"8038e00b-21c6-4743-9109-f7a60d1d5e32\") " pod="kuadrant-system/authorino-59c974bd48-2jncm"
Sep 18 15:43:59.015645 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:59.015624    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwghb\" (UniqueName: \"kubernetes.io/projected/8038e00b-21c6-4743-9109-f7a60d1d5e32-kube-api-access-cwghb\") pod \"authorino-59c974bd48-2jncm\" (UID: \"8038e00b-21c6-4743-9109-f7a60d1d5e32\") " pod="kuadrant-system/authorino-59c974bd48-2jncm"
Sep 18 15:43:59.106927 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:59.106849    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2jncm"
Sep 18 15:43:59.253170 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:59.253138    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-875ng"]
Sep 18 15:43:59.256643 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:43:59.256610    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecd4987a_3d8f_4ac6_9898_c9eb61be88fe.slice/crio-73e44035294e42de2d4a587f00f5695051a04205d541d46468747176885b8baa WatchSource:0}: Error finding container 73e44035294e42de2d4a587f00f5695051a04205d541d46468747176885b8baa: Status 404 returned error can't find the container with id 73e44035294e42de2d4a587f00f5695051a04205d541d46468747176885b8baa
Sep 18 15:43:59.376197 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:59.376171    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2jncm"]
Sep 18 15:43:59.379312 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:43:59.379282    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8038e00b_21c6_4743_9109_f7a60d1d5e32.slice/crio-edd734fdfc2395323bcad2f9cc6ca7d8b67885f5c82864cdfd15db3275bed481 WatchSource:0}: Error finding container edd734fdfc2395323bcad2f9cc6ca7d8b67885f5c82864cdfd15db3275bed481: Status 404 returned error can't find the container with id edd734fdfc2395323bcad2f9cc6ca7d8b67885f5c82864cdfd15db3275bed481
Sep 18 15:43:59.978426 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:59.978391    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-875ng" event={"ID":"ecd4987a-3d8f-4ac6-9898-c9eb61be88fe","Type":"ContainerStarted","Data":"73e44035294e42de2d4a587f00f5695051a04205d541d46468747176885b8baa"}
Sep 18 15:43:59.979440 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:43:59.979422    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2jncm" event={"ID":"8038e00b-21c6-4743-9109-f7a60d1d5e32","Type":"ContainerStarted","Data":"edd734fdfc2395323bcad2f9cc6ca7d8b67885f5c82864cdfd15db3275bed481"}
Sep 18 15:44:01.991551 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:01.991518    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2jncm" event={"ID":"8038e00b-21c6-4743-9109-f7a60d1d5e32","Type":"ContainerStarted","Data":"cff1a673062f64990cbeec20ac2c778b4d6afbec3e6a92deb63156f4cb2a0020"}
Sep 18 15:44:01.993055 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:01.993029    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-9cslp" event={"ID":"1943f126-8a84-429d-b8f0-270b9daeaf7e","Type":"ContainerStarted","Data":"17a34a29068d212a80cdb550000cd53a4a17318bfb8b7160b1508ed032c0db9a"}
Sep 18 15:44:01.993180 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:01.993112    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-9cslp"
Sep 18 15:44:01.994455 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:01.994435    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-875ng" event={"ID":"ecd4987a-3d8f-4ac6-9898-c9eb61be88fe","Type":"ContainerStarted","Data":"cfd2547b9725211a4e072e6dff1eb78f457eb253d6d3c66c20c0ac1afb1cae9a"}
Sep 18 15:44:02.008400 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:02.008358    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-2jncm" podStartSLOduration=1.816175903 podStartE2EDuration="4.00834483s" totalImagesPullingTime="2.192168927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:44:02.008026455 +0000 UTC m=+585.553317142" watchObservedRunningTime="2026-09-18 15:44:02.00834483 +0000 UTC m=+585.553635537"
Sep 18 15:44:02.026982 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:02.026940    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-9cslp" podStartSLOduration=1.936936336 podStartE2EDuration="5.026927834s" totalImagesPullingTime="3.089991498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:43:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:44:02.025312687 +0000 UTC m=+585.570603362" watchObservedRunningTime="2026-09-18 15:44:02.026927834 +0000 UTC m=+585.572218508"
Sep 18 15:44:02.037003 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:02.036976    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-875ng"]
Sep 18 15:44:02.054585 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:02.054539    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-875ng" podStartSLOduration=1.6304740789999999 podStartE2EDuration="4.054526108s" totalImagesPullingTime="2.424052029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:44:02.053720679 +0000 UTC m=+585.599011353" watchObservedRunningTime="2026-09-18 15:44:02.054526108 +0000 UTC m=+585.599816781"
Sep 18 15:44:04.004167 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:04.004129    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-875ng" podUID="ecd4987a-3d8f-4ac6-9898-c9eb61be88fe" containerName="authorino" containerID="cri-o://cfd2547b9725211a4e072e6dff1eb78f457eb253d6d3c66c20c0ac1afb1cae9a" gracePeriod=30
Sep 18 15:44:04.252239 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:04.252213    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-875ng"
Sep 18 15:44:04.352865 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:04.352794    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wsjb\" (UniqueName: \"kubernetes.io/projected/ecd4987a-3d8f-4ac6-9898-c9eb61be88fe-kube-api-access-6wsjb\") pod \"ecd4987a-3d8f-4ac6-9898-c9eb61be88fe\" (UID: \"ecd4987a-3d8f-4ac6-9898-c9eb61be88fe\") "
Sep 18 15:44:04.354989 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:04.354963    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd4987a-3d8f-4ac6-9898-c9eb61be88fe-kube-api-access-6wsjb" (OuterVolumeSpecName: "kube-api-access-6wsjb") pod "ecd4987a-3d8f-4ac6-9898-c9eb61be88fe" (UID: "ecd4987a-3d8f-4ac6-9898-c9eb61be88fe"). InnerVolumeSpecName "kube-api-access-6wsjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:44:04.453578 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:04.453552    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wsjb\" (UniqueName: \"kubernetes.io/projected/ecd4987a-3d8f-4ac6-9898-c9eb61be88fe-kube-api-access-6wsjb\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:44:05.009922 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:05.009885    2561 generic.go:356] "Generic (PLEG): container finished" podID="ecd4987a-3d8f-4ac6-9898-c9eb61be88fe" containerID="cfd2547b9725211a4e072e6dff1eb78f457eb253d6d3c66c20c0ac1afb1cae9a" exitCode=0
Sep 18 15:44:05.009922 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:05.009925    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-875ng"
Sep 18 15:44:05.010411 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:05.009970    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-875ng" event={"ID":"ecd4987a-3d8f-4ac6-9898-c9eb61be88fe","Type":"ContainerDied","Data":"cfd2547b9725211a4e072e6dff1eb78f457eb253d6d3c66c20c0ac1afb1cae9a"}
Sep 18 15:44:05.010411 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:05.010018    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-875ng" event={"ID":"ecd4987a-3d8f-4ac6-9898-c9eb61be88fe","Type":"ContainerDied","Data":"73e44035294e42de2d4a587f00f5695051a04205d541d46468747176885b8baa"}
Sep 18 15:44:05.010411 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:05.010040    2561 scope.go:117] "RemoveContainer" containerID="cfd2547b9725211a4e072e6dff1eb78f457eb253d6d3c66c20c0ac1afb1cae9a"
Sep 18 15:44:05.018246 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:05.018230    2561 scope.go:117] "RemoveContainer" containerID="cfd2547b9725211a4e072e6dff1eb78f457eb253d6d3c66c20c0ac1afb1cae9a"
Sep 18 15:44:05.018510 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:44:05.018495    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfd2547b9725211a4e072e6dff1eb78f457eb253d6d3c66c20c0ac1afb1cae9a\": container with ID starting with cfd2547b9725211a4e072e6dff1eb78f457eb253d6d3c66c20c0ac1afb1cae9a not found: ID does not exist" containerID="cfd2547b9725211a4e072e6dff1eb78f457eb253d6d3c66c20c0ac1afb1cae9a"
Sep 18 15:44:05.018562 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:05.018519    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfd2547b9725211a4e072e6dff1eb78f457eb253d6d3c66c20c0ac1afb1cae9a"} err="failed to get container status \"cfd2547b9725211a4e072e6dff1eb78f457eb253d6d3c66c20c0ac1afb1cae9a\": rpc error: code = NotFound desc = could not find container \"cfd2547b9725211a4e072e6dff1eb78f457eb253d6d3c66c20c0ac1afb1cae9a\": container with ID starting with cfd2547b9725211a4e072e6dff1eb78f457eb253d6d3c66c20c0ac1afb1cae9a not found: ID does not exist"
Sep 18 15:44:05.026737 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:05.026715    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-875ng"]
Sep 18 15:44:05.030767 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:05.030748    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-875ng"]
Sep 18 15:44:06.379789 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:06.379758    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecd4987a-3d8f-4ac6-9898-c9eb61be88fe" path="/var/lib/kubelet/pods/ecd4987a-3d8f-4ac6-9898-c9eb61be88fe/volumes"
Sep 18 15:44:11.026500 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:11.026468    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9cslp"]
Sep 18 15:44:11.026958 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:11.026711    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-9cslp" podUID="1943f126-8a84-429d-b8f0-270b9daeaf7e" containerName="limitador" containerID="cri-o://17a34a29068d212a80cdb550000cd53a4a17318bfb8b7160b1508ed032c0db9a" gracePeriod=30
Sep 18 15:44:11.027390 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:11.027278    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-9cslp"
Sep 18 15:44:11.574154 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:11.574132    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-9cslp"
Sep 18 15:44:11.713559 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:11.713535    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z9fb\" (UniqueName: \"kubernetes.io/projected/1943f126-8a84-429d-b8f0-270b9daeaf7e-kube-api-access-2z9fb\") pod \"1943f126-8a84-429d-b8f0-270b9daeaf7e\" (UID: \"1943f126-8a84-429d-b8f0-270b9daeaf7e\") "
Sep 18 15:44:11.713721 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:11.713584    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1943f126-8a84-429d-b8f0-270b9daeaf7e-config-file\") pod \"1943f126-8a84-429d-b8f0-270b9daeaf7e\" (UID: \"1943f126-8a84-429d-b8f0-270b9daeaf7e\") "
Sep 18 15:44:11.713969 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:11.713947    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1943f126-8a84-429d-b8f0-270b9daeaf7e-config-file" (OuterVolumeSpecName: "config-file") pod "1943f126-8a84-429d-b8f0-270b9daeaf7e" (UID: "1943f126-8a84-429d-b8f0-270b9daeaf7e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:44:11.715570 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:11.715545    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1943f126-8a84-429d-b8f0-270b9daeaf7e-kube-api-access-2z9fb" (OuterVolumeSpecName: "kube-api-access-2z9fb") pod "1943f126-8a84-429d-b8f0-270b9daeaf7e" (UID: "1943f126-8a84-429d-b8f0-270b9daeaf7e"). InnerVolumeSpecName "kube-api-access-2z9fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:44:11.815046 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:11.815022    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z9fb\" (UniqueName: \"kubernetes.io/projected/1943f126-8a84-429d-b8f0-270b9daeaf7e-kube-api-access-2z9fb\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:44:11.815046 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:11.815043    2561 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1943f126-8a84-429d-b8f0-270b9daeaf7e-config-file\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:44:12.051196 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:12.051120    2561 generic.go:356] "Generic (PLEG): container finished" podID="1943f126-8a84-429d-b8f0-270b9daeaf7e" containerID="17a34a29068d212a80cdb550000cd53a4a17318bfb8b7160b1508ed032c0db9a" exitCode=0
Sep 18 15:44:12.051196 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:12.051162    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-9cslp"
Sep 18 15:44:12.051756 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:12.051210    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-9cslp" event={"ID":"1943f126-8a84-429d-b8f0-270b9daeaf7e","Type":"ContainerDied","Data":"17a34a29068d212a80cdb550000cd53a4a17318bfb8b7160b1508ed032c0db9a"}
Sep 18 15:44:12.051756 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:12.051266    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-9cslp" event={"ID":"1943f126-8a84-429d-b8f0-270b9daeaf7e","Type":"ContainerDied","Data":"ed226e5cc59b9855d91bd0a4d9d01621513d953ecccd2d2bedbea213b3aa547e"}
Sep 18 15:44:12.051756 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:12.051298    2561 scope.go:117] "RemoveContainer" containerID="17a34a29068d212a80cdb550000cd53a4a17318bfb8b7160b1508ed032c0db9a"
Sep 18 15:44:12.061192 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:12.061177    2561 scope.go:117] "RemoveContainer" containerID="17a34a29068d212a80cdb550000cd53a4a17318bfb8b7160b1508ed032c0db9a"
Sep 18 15:44:12.061469 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:44:12.061452    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17a34a29068d212a80cdb550000cd53a4a17318bfb8b7160b1508ed032c0db9a\": container with ID starting with 17a34a29068d212a80cdb550000cd53a4a17318bfb8b7160b1508ed032c0db9a not found: ID does not exist" containerID="17a34a29068d212a80cdb550000cd53a4a17318bfb8b7160b1508ed032c0db9a"
Sep 18 15:44:12.061546 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:12.061476    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17a34a29068d212a80cdb550000cd53a4a17318bfb8b7160b1508ed032c0db9a"} err="failed to get container status \"17a34a29068d212a80cdb550000cd53a4a17318bfb8b7160b1508ed032c0db9a\": rpc error: code = NotFound desc = could not find container \"17a34a29068d212a80cdb550000cd53a4a17318bfb8b7160b1508ed032c0db9a\": container with ID starting with 17a34a29068d212a80cdb550000cd53a4a17318bfb8b7160b1508ed032c0db9a not found: ID does not exist"
Sep 18 15:44:12.074840 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:12.074816    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9cslp"]
Sep 18 15:44:12.078740 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:12.078721    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-9cslp"]
Sep 18 15:44:12.385422 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:12.385344    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1943f126-8a84-429d-b8f0-270b9daeaf7e" path="/var/lib/kubelet/pods/1943f126-8a84-429d-b8f0-270b9daeaf7e/volumes"
Sep 18 15:44:15.896927 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:15.896896    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh"]
Sep 18 15:44:15.897365 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:15.897221    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecd4987a-3d8f-4ac6-9898-c9eb61be88fe" containerName="authorino"
Sep 18 15:44:15.897365 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:15.897232    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecd4987a-3d8f-4ac6-9898-c9eb61be88fe" containerName="authorino"
Sep 18 15:44:15.897365 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:15.897261    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1943f126-8a84-429d-b8f0-270b9daeaf7e" containerName="limitador"
Sep 18 15:44:15.897365 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:15.897266    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1943f126-8a84-429d-b8f0-270b9daeaf7e" containerName="limitador"
Sep 18 15:44:15.897365 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:15.897335    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1943f126-8a84-429d-b8f0-270b9daeaf7e" containerName="limitador"
Sep 18 15:44:15.897365 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:15.897343    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecd4987a-3d8f-4ac6-9898-c9eb61be88fe" containerName="authorino"
Sep 18 15:44:15.901467 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:15.901448    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh"
Sep 18 15:44:15.904276 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:15.904255    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 15:44:15.904372 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:15.904308    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 18 15:44:15.904554 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:15.904542    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fzx6c\""
Sep 18 15:44:15.905235 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:15.905219    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 15:44:15.909092 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:15.908803    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh"]
Sep 18 15:44:16.055187 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:16.055157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35783dac-08df-497e-a857-7d762f4badda-data\") pod \"postgres-69fb4594cf-gfjlh\" (UID: \"35783dac-08df-497e-a857-7d762f4badda\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh"
Sep 18 15:44:16.055347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:16.055199    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh94b\" (UniqueName: \"kubernetes.io/projected/35783dac-08df-497e-a857-7d762f4badda-kube-api-access-fh94b\") pod \"postgres-69fb4594cf-gfjlh\" (UID: \"35783dac-08df-497e-a857-7d762f4badda\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh"
Sep 18 15:44:16.156590 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:16.156504    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh94b\" (UniqueName: \"kubernetes.io/projected/35783dac-08df-497e-a857-7d762f4badda-kube-api-access-fh94b\") pod \"postgres-69fb4594cf-gfjlh\" (UID: \"35783dac-08df-497e-a857-7d762f4badda\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh"
Sep 18 15:44:16.156750 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:16.156595    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35783dac-08df-497e-a857-7d762f4badda-data\") pod \"postgres-69fb4594cf-gfjlh\" (UID: \"35783dac-08df-497e-a857-7d762f4badda\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh"
Sep 18 15:44:16.156957 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:16.156935    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35783dac-08df-497e-a857-7d762f4badda-data\") pod \"postgres-69fb4594cf-gfjlh\" (UID: \"35783dac-08df-497e-a857-7d762f4badda\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh"
Sep 18 15:44:16.167312 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:16.167286    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh94b\" (UniqueName: \"kubernetes.io/projected/35783dac-08df-497e-a857-7d762f4badda-kube-api-access-fh94b\") pod \"postgres-69fb4594cf-gfjlh\" (UID: \"35783dac-08df-497e-a857-7d762f4badda\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh"
Sep 18 15:44:16.213036 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:16.212993    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh"
Sep 18 15:44:16.509631 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:16.509595    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh"]
Sep 18 15:44:16.510440 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:44:16.510414    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35783dac_08df_497e_a857_7d762f4badda.slice/crio-dae8d830edbdacd3d8b08a8e34069e5c11bc2ce7a0b7956eedee1c628e458e7f WatchSource:0}: Error finding container dae8d830edbdacd3d8b08a8e34069e5c11bc2ce7a0b7956eedee1c628e458e7f: Status 404 returned error can't find the container with id dae8d830edbdacd3d8b08a8e34069e5c11bc2ce7a0b7956eedee1c628e458e7f
Sep 18 15:44:17.075947 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:17.075909    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh" event={"ID":"35783dac-08df-497e-a857-7d762f4badda","Type":"ContainerStarted","Data":"dae8d830edbdacd3d8b08a8e34069e5c11bc2ce7a0b7956eedee1c628e458e7f"}
Sep 18 15:44:18.313577 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:18.313549    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 15:44:18.314555 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:18.314523    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 15:44:21.304393 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:21.304369    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 18 15:44:22.102071 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:22.102039    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh" event={"ID":"35783dac-08df-497e-a857-7d762f4badda","Type":"ContainerStarted","Data":"ffab9c1664a6f4a938415e10a33811d1f5c9e4447de2b6f89fbafff4af661c2a"}
Sep 18 15:44:22.102274 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:22.102121    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh"
Sep 18 15:44:22.120423 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:22.120369    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh" podStartSLOduration=2.3303687699999998 podStartE2EDuration="7.120349958s" totalImagesPullingTime="4.789981188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:44:22.119017542 +0000 UTC m=+605.664308217" watchObservedRunningTime="2026-09-18 15:44:22.120349958 +0000 UTC m=+605.665640633"
Sep 18 15:44:28.135847 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.135818    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gfjlh"
Sep 18 15:44:28.686628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.686592    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-xmdww"]
Sep 18 15:44:28.692507 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.692489    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xmdww"
Sep 18 15:44:28.700716 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.700694    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-xmdww"]
Sep 18 15:44:28.780337 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.780310    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-xmdww"]
Sep 18 15:44:28.780538 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:44:28.780520    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-8p4hc], unattached volumes=[], failed to process volumes=[kube-api-access-8p4hc]: context canceled" pod="kuadrant-system/authorino-557f876f69-xmdww" podUID="9a7483bb-b984-4b45-8d0b-35f1b5da962f"
Sep 18 15:44:28.807059 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.807029    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8gmxw"]
Sep 18 15:44:28.810744 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.810723    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-8gmxw"
Sep 18 15:44:28.813098 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.813081    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 18 15:44:28.819019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.818991    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8gmxw"]
Sep 18 15:44:28.879582 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.879552    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p4hc\" (UniqueName: \"kubernetes.io/projected/9a7483bb-b984-4b45-8d0b-35f1b5da962f-kube-api-access-8p4hc\") pod \"authorino-557f876f69-xmdww\" (UID: \"9a7483bb-b984-4b45-8d0b-35f1b5da962f\") " pod="kuadrant-system/authorino-557f876f69-xmdww"
Sep 18 15:44:28.931591 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.931557    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8gmxw"]
Sep 18 15:44:28.931853 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:44:28.931832    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-c6skd tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-8gmxw" podUID="9fe06f71-4f12-4eda-881b-dbf342d14bb5"
Sep 18 15:44:28.964162 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.964100    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b7fcd96f8-mld8l"]
Sep 18 15:44:28.971085 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.971068    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b7fcd96f8-mld8l"
Sep 18 15:44:28.975262 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.975235    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b7fcd96f8-mld8l"]
Sep 18 15:44:28.980538 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.980520    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p4hc\" (UniqueName: \"kubernetes.io/projected/9a7483bb-b984-4b45-8d0b-35f1b5da962f-kube-api-access-8p4hc\") pod \"authorino-557f876f69-xmdww\" (UID: \"9a7483bb-b984-4b45-8d0b-35f1b5da962f\") " pod="kuadrant-system/authorino-557f876f69-xmdww"
Sep 18 15:44:28.980650 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.980546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6skd\" (UniqueName: \"kubernetes.io/projected/9fe06f71-4f12-4eda-881b-dbf342d14bb5-kube-api-access-c6skd\") pod \"authorino-6d4dfc649f-8gmxw\" (UID: \"9fe06f71-4f12-4eda-881b-dbf342d14bb5\") " pod="kuadrant-system/authorino-6d4dfc649f-8gmxw"
Sep 18 15:44:28.980650 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.980580    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fe06f71-4f12-4eda-881b-dbf342d14bb5-tls-cert\") pod \"authorino-6d4dfc649f-8gmxw\" (UID: \"9fe06f71-4f12-4eda-881b-dbf342d14bb5\") " pod="kuadrant-system/authorino-6d4dfc649f-8gmxw"
Sep 18 15:44:28.990409 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:28.990386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p4hc\" (UniqueName: \"kubernetes.io/projected/9a7483bb-b984-4b45-8d0b-35f1b5da962f-kube-api-access-8p4hc\") pod \"authorino-557f876f69-xmdww\" (UID: \"9a7483bb-b984-4b45-8d0b-35f1b5da962f\") " pod="kuadrant-system/authorino-557f876f69-xmdww"
Sep 18 15:44:29.081976 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.081948    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7m7\" (UniqueName: \"kubernetes.io/projected/ee4eae1a-0d12-4a97-bc66-652aa2f66c18-kube-api-access-9s7m7\") pod \"authorino-b7fcd96f8-mld8l\" (UID: \"ee4eae1a-0d12-4a97-bc66-652aa2f66c18\") " pod="kuadrant-system/authorino-b7fcd96f8-mld8l"
Sep 18 15:44:29.082130 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.081994    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6skd\" (UniqueName: \"kubernetes.io/projected/9fe06f71-4f12-4eda-881b-dbf342d14bb5-kube-api-access-c6skd\") pod \"authorino-6d4dfc649f-8gmxw\" (UID: \"9fe06f71-4f12-4eda-881b-dbf342d14bb5\") " pod="kuadrant-system/authorino-6d4dfc649f-8gmxw"
Sep 18 15:44:29.082130 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.082020    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee4eae1a-0d12-4a97-bc66-652aa2f66c18-tls-cert\") pod \"authorino-b7fcd96f8-mld8l\" (UID: \"ee4eae1a-0d12-4a97-bc66-652aa2f66c18\") " pod="kuadrant-system/authorino-b7fcd96f8-mld8l"
Sep 18 15:44:29.082238 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.082141    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fe06f71-4f12-4eda-881b-dbf342d14bb5-tls-cert\") pod \"authorino-6d4dfc649f-8gmxw\" (UID: \"9fe06f71-4f12-4eda-881b-dbf342d14bb5\") " pod="kuadrant-system/authorino-6d4dfc649f-8gmxw"
Sep 18 15:44:29.084564 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.084545    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fe06f71-4f12-4eda-881b-dbf342d14bb5-tls-cert\") pod \"authorino-6d4dfc649f-8gmxw\" (UID: \"9fe06f71-4f12-4eda-881b-dbf342d14bb5\") " pod="kuadrant-system/authorino-6d4dfc649f-8gmxw"
Sep 18 15:44:29.089987 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.089970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6skd\" (UniqueName: \"kubernetes.io/projected/9fe06f71-4f12-4eda-881b-dbf342d14bb5-kube-api-access-c6skd\") pod \"authorino-6d4dfc649f-8gmxw\" (UID: \"9fe06f71-4f12-4eda-881b-dbf342d14bb5\") " pod="kuadrant-system/authorino-6d4dfc649f-8gmxw"
Sep 18 15:44:29.135085 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.135061    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xmdww"
Sep 18 15:44:29.135220 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.135064    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-8gmxw"
Sep 18 15:44:29.146436 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.146417    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xmdww"
Sep 18 15:44:29.149394 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.149376    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-8gmxw"
Sep 18 15:44:29.183043 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.183020    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7m7\" (UniqueName: \"kubernetes.io/projected/ee4eae1a-0d12-4a97-bc66-652aa2f66c18-kube-api-access-9s7m7\") pod \"authorino-b7fcd96f8-mld8l\" (UID: \"ee4eae1a-0d12-4a97-bc66-652aa2f66c18\") " pod="kuadrant-system/authorino-b7fcd96f8-mld8l"
Sep 18 15:44:29.183172 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.183066    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee4eae1a-0d12-4a97-bc66-652aa2f66c18-tls-cert\") pod \"authorino-b7fcd96f8-mld8l\" (UID: \"ee4eae1a-0d12-4a97-bc66-652aa2f66c18\") " pod="kuadrant-system/authorino-b7fcd96f8-mld8l"
Sep 18 15:44:29.185426 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.185388    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee4eae1a-0d12-4a97-bc66-652aa2f66c18-tls-cert\") pod \"authorino-b7fcd96f8-mld8l\" (UID: \"ee4eae1a-0d12-4a97-bc66-652aa2f66c18\") " pod="kuadrant-system/authorino-b7fcd96f8-mld8l"
Sep 18 15:44:29.191267 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.191248    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7m7\" (UniqueName: \"kubernetes.io/projected/ee4eae1a-0d12-4a97-bc66-652aa2f66c18-kube-api-access-9s7m7\") pod \"authorino-b7fcd96f8-mld8l\" (UID: \"ee4eae1a-0d12-4a97-bc66-652aa2f66c18\") " pod="kuadrant-system/authorino-b7fcd96f8-mld8l"
Sep 18 15:44:29.281316 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.281253    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b7fcd96f8-mld8l"
Sep 18 15:44:29.283525 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.283502    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fe06f71-4f12-4eda-881b-dbf342d14bb5-tls-cert\") pod \"9fe06f71-4f12-4eda-881b-dbf342d14bb5\" (UID: \"9fe06f71-4f12-4eda-881b-dbf342d14bb5\") "
Sep 18 15:44:29.283633 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.283577    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p4hc\" (UniqueName: \"kubernetes.io/projected/9a7483bb-b984-4b45-8d0b-35f1b5da962f-kube-api-access-8p4hc\") pod \"9a7483bb-b984-4b45-8d0b-35f1b5da962f\" (UID: \"9a7483bb-b984-4b45-8d0b-35f1b5da962f\") "
Sep 18 15:44:29.283706 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.283642    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6skd\" (UniqueName: \"kubernetes.io/projected/9fe06f71-4f12-4eda-881b-dbf342d14bb5-kube-api-access-c6skd\") pod \"9fe06f71-4f12-4eda-881b-dbf342d14bb5\" (UID: \"9fe06f71-4f12-4eda-881b-dbf342d14bb5\") "
Sep 18 15:44:29.285604 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.285580    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe06f71-4f12-4eda-881b-dbf342d14bb5-kube-api-access-c6skd" (OuterVolumeSpecName: "kube-api-access-c6skd") pod "9fe06f71-4f12-4eda-881b-dbf342d14bb5" (UID: "9fe06f71-4f12-4eda-881b-dbf342d14bb5"). InnerVolumeSpecName "kube-api-access-c6skd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:44:29.285712 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.285613    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe06f71-4f12-4eda-881b-dbf342d14bb5-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "9fe06f71-4f12-4eda-881b-dbf342d14bb5" (UID: "9fe06f71-4f12-4eda-881b-dbf342d14bb5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:44:29.285712 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.285654    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7483bb-b984-4b45-8d0b-35f1b5da962f-kube-api-access-8p4hc" (OuterVolumeSpecName: "kube-api-access-8p4hc") pod "9a7483bb-b984-4b45-8d0b-35f1b5da962f" (UID: "9a7483bb-b984-4b45-8d0b-35f1b5da962f"). InnerVolumeSpecName "kube-api-access-8p4hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:44:29.384573 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.384532    2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9fe06f71-4f12-4eda-881b-dbf342d14bb5-tls-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:44:29.384698 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.384580    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p4hc\" (UniqueName: \"kubernetes.io/projected/9a7483bb-b984-4b45-8d0b-35f1b5da962f-kube-api-access-8p4hc\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:44:29.384698 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.384597    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6skd\" (UniqueName: \"kubernetes.io/projected/9fe06f71-4f12-4eda-881b-dbf342d14bb5-kube-api-access-c6skd\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:44:29.587529 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:29.587475    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b7fcd96f8-mld8l"]
Sep 18 15:44:29.589862 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:44:29.589839    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee4eae1a_0d12_4a97_bc66_652aa2f66c18.slice/crio-221313a5b6924e79c7f3d7da65b5cdb627ef382c9083a144ac7a9c418eb6fe2a WatchSource:0}: Error finding container 221313a5b6924e79c7f3d7da65b5cdb627ef382c9083a144ac7a9c418eb6fe2a: Status 404 returned error can't find the container with id 221313a5b6924e79c7f3d7da65b5cdb627ef382c9083a144ac7a9c418eb6fe2a
Sep 18 15:44:30.141269 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:30.141231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b7fcd96f8-mld8l" event={"ID":"ee4eae1a-0d12-4a97-bc66-652aa2f66c18","Type":"ContainerStarted","Data":"221313a5b6924e79c7f3d7da65b5cdb627ef382c9083a144ac7a9c418eb6fe2a"}
Sep 18 15:44:30.141269 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:30.141257    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-8gmxw"
Sep 18 15:44:30.141269 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:30.141271    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-xmdww"
Sep 18 15:44:30.172228 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:30.172202    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-xmdww"]
Sep 18 15:44:30.174124 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:30.174104    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-xmdww"]
Sep 18 15:44:30.193861 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:30.193838    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8gmxw"]
Sep 18 15:44:30.197844 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:30.197825    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-8gmxw"]
Sep 18 15:44:30.380965 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:30.380900    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a7483bb-b984-4b45-8d0b-35f1b5da962f" path="/var/lib/kubelet/pods/9a7483bb-b984-4b45-8d0b-35f1b5da962f/volumes"
Sep 18 15:44:30.381183 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:30.381171    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fe06f71-4f12-4eda-881b-dbf342d14bb5" path="/var/lib/kubelet/pods/9fe06f71-4f12-4eda-881b-dbf342d14bb5/volumes"
Sep 18 15:44:31.147580 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:31.147550    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b7fcd96f8-mld8l" event={"ID":"ee4eae1a-0d12-4a97-bc66-652aa2f66c18","Type":"ContainerStarted","Data":"95a65cdde10454f001bde5222d416c3c05d100db806cdfad74f84b319d38d9d5"}
Sep 18 15:44:31.162812 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:31.162771    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b7fcd96f8-mld8l" podStartSLOduration=2.178334997 podStartE2EDuration="3.162755392s" totalImagesPullingTime="984.420395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:44:31.162020078 +0000 UTC m=+614.707310754" watchObservedRunningTime="2026-09-18 15:44:31.162755392 +0000 UTC m=+614.708046066"
Sep 18 15:44:31.190897 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:31.190867    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2jncm"]
Sep 18 15:44:31.191275 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:31.191067    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-2jncm" podUID="8038e00b-21c6-4743-9109-f7a60d1d5e32" containerName="authorino" containerID="cri-o://cff1a673062f64990cbeec20ac2c778b4d6afbec3e6a92deb63156f4cb2a0020" gracePeriod=30
Sep 18 15:44:31.473111 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:31.473087    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2jncm"
Sep 18 15:44:31.503596 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:31.503569    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwghb\" (UniqueName: \"kubernetes.io/projected/8038e00b-21c6-4743-9109-f7a60d1d5e32-kube-api-access-cwghb\") pod \"8038e00b-21c6-4743-9109-f7a60d1d5e32\" (UID: \"8038e00b-21c6-4743-9109-f7a60d1d5e32\") "
Sep 18 15:44:31.505632 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:31.505608    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8038e00b-21c6-4743-9109-f7a60d1d5e32-kube-api-access-cwghb" (OuterVolumeSpecName: "kube-api-access-cwghb") pod "8038e00b-21c6-4743-9109-f7a60d1d5e32" (UID: "8038e00b-21c6-4743-9109-f7a60d1d5e32"). InnerVolumeSpecName "kube-api-access-cwghb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:44:31.604417 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:31.604391    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwghb\" (UniqueName: \"kubernetes.io/projected/8038e00b-21c6-4743-9109-f7a60d1d5e32-kube-api-access-cwghb\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:44:32.153648 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:32.153614    2561 generic.go:356] "Generic (PLEG): container finished" podID="8038e00b-21c6-4743-9109-f7a60d1d5e32" containerID="cff1a673062f64990cbeec20ac2c778b4d6afbec3e6a92deb63156f4cb2a0020" exitCode=0
Sep 18 15:44:32.153831 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:32.153655    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2jncm"
Sep 18 15:44:32.153831 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:32.153698    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2jncm" event={"ID":"8038e00b-21c6-4743-9109-f7a60d1d5e32","Type":"ContainerDied","Data":"cff1a673062f64990cbeec20ac2c778b4d6afbec3e6a92deb63156f4cb2a0020"}
Sep 18 15:44:32.153831 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:32.153742    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2jncm" event={"ID":"8038e00b-21c6-4743-9109-f7a60d1d5e32","Type":"ContainerDied","Data":"edd734fdfc2395323bcad2f9cc6ca7d8b67885f5c82864cdfd15db3275bed481"}
Sep 18 15:44:32.153831 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:32.153761    2561 scope.go:117] "RemoveContainer" containerID="cff1a673062f64990cbeec20ac2c778b4d6afbec3e6a92deb63156f4cb2a0020"
Sep 18 15:44:32.163208 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:32.163192    2561 scope.go:117] "RemoveContainer" containerID="cff1a673062f64990cbeec20ac2c778b4d6afbec3e6a92deb63156f4cb2a0020"
Sep 18 15:44:32.163542 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:44:32.163523    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff1a673062f64990cbeec20ac2c778b4d6afbec3e6a92deb63156f4cb2a0020\": container with ID starting with cff1a673062f64990cbeec20ac2c778b4d6afbec3e6a92deb63156f4cb2a0020 not found: ID does not exist" containerID="cff1a673062f64990cbeec20ac2c778b4d6afbec3e6a92deb63156f4cb2a0020"
Sep 18 15:44:32.163607 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:32.163550    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff1a673062f64990cbeec20ac2c778b4d6afbec3e6a92deb63156f4cb2a0020"} err="failed to get container status \"cff1a673062f64990cbeec20ac2c778b4d6afbec3e6a92deb63156f4cb2a0020\": rpc error: code = NotFound desc = could not find container \"cff1a673062f64990cbeec20ac2c778b4d6afbec3e6a92deb63156f4cb2a0020\": container with ID starting with cff1a673062f64990cbeec20ac2c778b4d6afbec3e6a92deb63156f4cb2a0020 not found: ID does not exist"
Sep 18 15:44:32.175139 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:32.175115    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2jncm"]
Sep 18 15:44:32.179265 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:32.179233    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2jncm"]
Sep 18 15:44:32.380568 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:32.380537    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8038e00b-21c6-4743-9109-f7a60d1d5e32" path="/var/lib/kubelet/pods/8038e00b-21c6-4743-9109-f7a60d1d5e32/volumes"
Sep 18 15:44:35.940758 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:35.940638    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-8rhfs"]
Sep 18 15:44:35.941158 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:35.941032    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8038e00b-21c6-4743-9109-f7a60d1d5e32" containerName="authorino"
Sep 18 15:44:35.941158 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:35.941045    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8038e00b-21c6-4743-9109-f7a60d1d5e32" containerName="authorino"
Sep 18 15:44:35.941158 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:35.941108    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8038e00b-21c6-4743-9109-f7a60d1d5e32" containerName="authorino"
Sep 18 15:44:35.945808 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:35.945791    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:35.948299 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:35.948278    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qjgkv\""
Sep 18 15:44:35.948631 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:35.948615    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 15:44:35.948912 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:35.948896    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 15:44:35.949294 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:35.949279    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 15:44:35.957518 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:35.957495    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-8rhfs"]
Sep 18 15:44:36.042055 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.042024    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-webhook-cert\") pod \"maas-controller-75dc6cbdf-8rhfs\" (UID: \"943cbf7b-ab51-4120-a404-1228e1637664\") " pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:36.042055 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.042057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-metrics-certs\") pod \"maas-controller-75dc6cbdf-8rhfs\" (UID: \"943cbf7b-ab51-4120-a404-1228e1637664\") " pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:36.042248 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.042144    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bprn9\" (UniqueName: \"kubernetes.io/projected/943cbf7b-ab51-4120-a404-1228e1637664-kube-api-access-bprn9\") pod \"maas-controller-75dc6cbdf-8rhfs\" (UID: \"943cbf7b-ab51-4120-a404-1228e1637664\") " pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:36.129247 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.129215    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-zmld8"]
Sep 18 15:44:36.132411 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.132396    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:44:36.142819 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.142797    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bprn9\" (UniqueName: \"kubernetes.io/projected/943cbf7b-ab51-4120-a404-1228e1637664-kube-api-access-bprn9\") pod \"maas-controller-75dc6cbdf-8rhfs\" (UID: \"943cbf7b-ab51-4120-a404-1228e1637664\") " pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:36.142925 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.142838    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-webhook-cert\") pod \"maas-controller-75dc6cbdf-8rhfs\" (UID: \"943cbf7b-ab51-4120-a404-1228e1637664\") " pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:36.142925 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.142859    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-metrics-certs\") pod \"maas-controller-75dc6cbdf-8rhfs\" (UID: \"943cbf7b-ab51-4120-a404-1228e1637664\") " pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:36.143003 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:44:36.142988    2561 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 18 15:44:36.143051 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:44:36.143046    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-metrics-certs podName:943cbf7b-ab51-4120-a404-1228e1637664 nodeName:}" failed. No retries permitted until 2026-09-18 15:44:36.643029053 +0000 UTC m=+620.188319719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-metrics-certs") pod "maas-controller-75dc6cbdf-8rhfs" (UID: "943cbf7b-ab51-4120-a404-1228e1637664") : secret "maas-controller-metrics-tls" not found
Sep 18 15:44:36.145422 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.145397    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-webhook-cert\") pod \"maas-controller-75dc6cbdf-8rhfs\" (UID: \"943cbf7b-ab51-4120-a404-1228e1637664\") " pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:36.147135 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.147109    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-zmld8"]
Sep 18 15:44:36.152302 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.152279    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bprn9\" (UniqueName: \"kubernetes.io/projected/943cbf7b-ab51-4120-a404-1228e1637664-kube-api-access-bprn9\") pod \"maas-controller-75dc6cbdf-8rhfs\" (UID: \"943cbf7b-ab51-4120-a404-1228e1637664\") " pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:36.243748 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.243651    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16a1987-43b8-4d38-b03f-f5da907d36ec-metrics-certs\") pod \"maas-controller-7f9bc5dc6f-zmld8\" (UID: \"c16a1987-43b8-4d38-b03f-f5da907d36ec\") " pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:44:36.243748 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.243697    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c16a1987-43b8-4d38-b03f-f5da907d36ec-webhook-cert\") pod \"maas-controller-7f9bc5dc6f-zmld8\" (UID: \"c16a1987-43b8-4d38-b03f-f5da907d36ec\") " pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:44:36.243920 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.243819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j7ss\" (UniqueName: \"kubernetes.io/projected/c16a1987-43b8-4d38-b03f-f5da907d36ec-kube-api-access-6j7ss\") pod \"maas-controller-7f9bc5dc6f-zmld8\" (UID: \"c16a1987-43b8-4d38-b03f-f5da907d36ec\") " pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:44:36.344797 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.344766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j7ss\" (UniqueName: \"kubernetes.io/projected/c16a1987-43b8-4d38-b03f-f5da907d36ec-kube-api-access-6j7ss\") pod \"maas-controller-7f9bc5dc6f-zmld8\" (UID: \"c16a1987-43b8-4d38-b03f-f5da907d36ec\") " pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:44:36.344797 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.344798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16a1987-43b8-4d38-b03f-f5da907d36ec-metrics-certs\") pod \"maas-controller-7f9bc5dc6f-zmld8\" (UID: \"c16a1987-43b8-4d38-b03f-f5da907d36ec\") " pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:44:36.345006 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.344817    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c16a1987-43b8-4d38-b03f-f5da907d36ec-webhook-cert\") pod \"maas-controller-7f9bc5dc6f-zmld8\" (UID: \"c16a1987-43b8-4d38-b03f-f5da907d36ec\") " pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:44:36.347402 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.347370    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c16a1987-43b8-4d38-b03f-f5da907d36ec-webhook-cert\") pod \"maas-controller-7f9bc5dc6f-zmld8\" (UID: \"c16a1987-43b8-4d38-b03f-f5da907d36ec\") " pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:44:36.347402 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.347386    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16a1987-43b8-4d38-b03f-f5da907d36ec-metrics-certs\") pod \"maas-controller-7f9bc5dc6f-zmld8\" (UID: \"c16a1987-43b8-4d38-b03f-f5da907d36ec\") " pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:44:36.353077 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.353059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j7ss\" (UniqueName: \"kubernetes.io/projected/c16a1987-43b8-4d38-b03f-f5da907d36ec-kube-api-access-6j7ss\") pod \"maas-controller-7f9bc5dc6f-zmld8\" (UID: \"c16a1987-43b8-4d38-b03f-f5da907d36ec\") " pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:44:36.442061 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.442036    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:44:36.648355 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.648266    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-metrics-certs\") pod \"maas-controller-75dc6cbdf-8rhfs\" (UID: \"943cbf7b-ab51-4120-a404-1228e1637664\") " pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:36.650719 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.650686    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-metrics-certs\") pod \"maas-controller-75dc6cbdf-8rhfs\" (UID: \"943cbf7b-ab51-4120-a404-1228e1637664\") " pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:36.711948 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.711925    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-zmld8"]
Sep 18 15:44:36.714056 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:44:36.714032    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16a1987_43b8_4d38_b03f_f5da907d36ec.slice/crio-679a96e3e403f9739d0753c4bf7864918b308f6cd4cd5a4333361cbcd98215fb WatchSource:0}: Error finding container 679a96e3e403f9739d0753c4bf7864918b308f6cd4cd5a4333361cbcd98215fb: Status 404 returned error can't find the container with id 679a96e3e403f9739d0753c4bf7864918b308f6cd4cd5a4333361cbcd98215fb
Sep 18 15:44:36.855979 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:36.855945    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:37.124361 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:37.124285    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-8rhfs"]
Sep 18 15:44:37.126543 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:44:37.126515    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943cbf7b_ab51_4120_a404_1228e1637664.slice/crio-d0fa838625a30c146ef5a1ff79d3e9f9874eff6a20b9b537990a74276f01ee76 WatchSource:0}: Error finding container d0fa838625a30c146ef5a1ff79d3e9f9874eff6a20b9b537990a74276f01ee76: Status 404 returned error can't find the container with id d0fa838625a30c146ef5a1ff79d3e9f9874eff6a20b9b537990a74276f01ee76
Sep 18 15:44:37.179566 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:37.179535    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-8rhfs" event={"ID":"943cbf7b-ab51-4120-a404-1228e1637664","Type":"ContainerStarted","Data":"d0fa838625a30c146ef5a1ff79d3e9f9874eff6a20b9b537990a74276f01ee76"}
Sep 18 15:44:37.181043 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:37.180995    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8" event={"ID":"c16a1987-43b8-4d38-b03f-f5da907d36ec","Type":"ContainerStarted","Data":"679a96e3e403f9739d0753c4bf7864918b308f6cd4cd5a4333361cbcd98215fb"}
Sep 18 15:44:40.199331 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:40.199284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8" event={"ID":"c16a1987-43b8-4d38-b03f-f5da907d36ec","Type":"ContainerStarted","Data":"a11a7024185df5c97a83c86a5f33cd92c5167714d03b8b695cdbc43b488f46a1"}
Sep 18 15:44:40.199805 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:40.199451    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:44:40.200800 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:40.200771    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-8rhfs" event={"ID":"943cbf7b-ab51-4120-a404-1228e1637664","Type":"ContainerStarted","Data":"3af3ca9ef2d6ea8f1052b537d4a10e8fd20853eb7aa654fe9e77c85add6be82e"}
Sep 18 15:44:40.200908 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:40.200899    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:40.220258 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:40.220207    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8" podStartSLOduration=1.744757626 podStartE2EDuration="4.220192884s" totalImagesPullingTime="2.475435258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:44:40.218883986 +0000 UTC m=+623.764174661" watchObservedRunningTime="2026-09-18 15:44:40.220192884 +0000 UTC m=+623.765483558"
Sep 18 15:44:40.237902 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:40.237858    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75dc6cbdf-8rhfs" podStartSLOduration=3.164686848 podStartE2EDuration="5.237843936s" totalImagesPullingTime="2.073157088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:44:40.237064658 +0000 UTC m=+623.782355333" watchObservedRunningTime="2026-09-18 15:44:40.237843936 +0000 UTC m=+623.783134610"
Sep 18 15:44:51.208151 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.208121    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:44:51.208682 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.208189    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:51.279562 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.279522    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-8rhfs"]
Sep 18 15:44:51.279853 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.279805    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75dc6cbdf-8rhfs" podUID="943cbf7b-ab51-4120-a404-1228e1637664" containerName="manager" containerID="cri-o://3af3ca9ef2d6ea8f1052b537d4a10e8fd20853eb7aa654fe9e77c85add6be82e" gracePeriod=30
Sep 18 15:44:51.521419 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.521394    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:51.577232 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.577208    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-metrics-certs\") pod \"943cbf7b-ab51-4120-a404-1228e1637664\" (UID: \"943cbf7b-ab51-4120-a404-1228e1637664\") "
Sep 18 15:44:51.577392 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.577243    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-webhook-cert\") pod \"943cbf7b-ab51-4120-a404-1228e1637664\" (UID: \"943cbf7b-ab51-4120-a404-1228e1637664\") "
Sep 18 15:44:51.577392 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.577287    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bprn9\" (UniqueName: \"kubernetes.io/projected/943cbf7b-ab51-4120-a404-1228e1637664-kube-api-access-bprn9\") pod \"943cbf7b-ab51-4120-a404-1228e1637664\" (UID: \"943cbf7b-ab51-4120-a404-1228e1637664\") "
Sep 18 15:44:51.579436 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.579408    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "943cbf7b-ab51-4120-a404-1228e1637664" (UID: "943cbf7b-ab51-4120-a404-1228e1637664"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:44:51.579538 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.579463    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "943cbf7b-ab51-4120-a404-1228e1637664" (UID: "943cbf7b-ab51-4120-a404-1228e1637664"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:44:51.579538 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.579477    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943cbf7b-ab51-4120-a404-1228e1637664-kube-api-access-bprn9" (OuterVolumeSpecName: "kube-api-access-bprn9") pod "943cbf7b-ab51-4120-a404-1228e1637664" (UID: "943cbf7b-ab51-4120-a404-1228e1637664"). InnerVolumeSpecName "kube-api-access-bprn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:44:51.678285 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.678246    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bprn9\" (UniqueName: \"kubernetes.io/projected/943cbf7b-ab51-4120-a404-1228e1637664-kube-api-access-bprn9\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:44:51.678285 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.678283    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-metrics-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:44:51.678285 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:51.678293    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/943cbf7b-ab51-4120-a404-1228e1637664-webhook-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:44:52.264233 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:52.264196    2561 generic.go:356] "Generic (PLEG): container finished" podID="943cbf7b-ab51-4120-a404-1228e1637664" containerID="3af3ca9ef2d6ea8f1052b537d4a10e8fd20853eb7aa654fe9e77c85add6be82e" exitCode=0
Sep 18 15:44:52.264614 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:52.264246    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75dc6cbdf-8rhfs"
Sep 18 15:44:52.264614 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:52.264285    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-8rhfs" event={"ID":"943cbf7b-ab51-4120-a404-1228e1637664","Type":"ContainerDied","Data":"3af3ca9ef2d6ea8f1052b537d4a10e8fd20853eb7aa654fe9e77c85add6be82e"}
Sep 18 15:44:52.264614 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:52.264339    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75dc6cbdf-8rhfs" event={"ID":"943cbf7b-ab51-4120-a404-1228e1637664","Type":"ContainerDied","Data":"d0fa838625a30c146ef5a1ff79d3e9f9874eff6a20b9b537990a74276f01ee76"}
Sep 18 15:44:52.264614 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:52.264368    2561 scope.go:117] "RemoveContainer" containerID="3af3ca9ef2d6ea8f1052b537d4a10e8fd20853eb7aa654fe9e77c85add6be82e"
Sep 18 15:44:52.273241 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:52.273219    2561 scope.go:117] "RemoveContainer" containerID="3af3ca9ef2d6ea8f1052b537d4a10e8fd20853eb7aa654fe9e77c85add6be82e"
Sep 18 15:44:52.273496 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:44:52.273478    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3af3ca9ef2d6ea8f1052b537d4a10e8fd20853eb7aa654fe9e77c85add6be82e\": container with ID starting with 3af3ca9ef2d6ea8f1052b537d4a10e8fd20853eb7aa654fe9e77c85add6be82e not found: ID does not exist" containerID="3af3ca9ef2d6ea8f1052b537d4a10e8fd20853eb7aa654fe9e77c85add6be82e"
Sep 18 15:44:52.273545 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:52.273502    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3af3ca9ef2d6ea8f1052b537d4a10e8fd20853eb7aa654fe9e77c85add6be82e"} err="failed to get container status \"3af3ca9ef2d6ea8f1052b537d4a10e8fd20853eb7aa654fe9e77c85add6be82e\": rpc error: code = NotFound desc = could not find container \"3af3ca9ef2d6ea8f1052b537d4a10e8fd20853eb7aa654fe9e77c85add6be82e\": container with ID starting with 3af3ca9ef2d6ea8f1052b537d4a10e8fd20853eb7aa654fe9e77c85add6be82e not found: ID does not exist"
Sep 18 15:44:52.288137 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:52.288118    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-8rhfs"]
Sep 18 15:44:52.290138 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:52.290114    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75dc6cbdf-8rhfs"]
Sep 18 15:44:52.379770 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:44:52.379749    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="943cbf7b-ab51-4120-a404-1228e1637664" path="/var/lib/kubelet/pods/943cbf7b-ab51-4120-a404-1228e1637664/volumes"
Sep 18 15:45:14.313079 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.313039    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"]
Sep 18 15:45:14.316812 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.313710    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="943cbf7b-ab51-4120-a404-1228e1637664" containerName="manager"
Sep 18 15:45:14.316812 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.313731    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="943cbf7b-ab51-4120-a404-1228e1637664" containerName="manager"
Sep 18 15:45:14.316812 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.313835    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="943cbf7b-ab51-4120-a404-1228e1637664" containerName="manager"
Sep 18 15:45:14.316812 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.316767    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"
Sep 18 15:45:14.320438 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.320412    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-gnptn\""
Sep 18 15:45:14.320853 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.320835    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 18 15:45:14.333002 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.332976    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"]
Sep 18 15:45:14.472710 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.472644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512e484b-c79d-4df3-bf9a-0502ed5bd338-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-t9h6w\" (UID: \"512e484b-c79d-4df3-bf9a-0502ed5bd338\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"
Sep 18 15:45:14.472906 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.472854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8cg9\" (UniqueName: \"kubernetes.io/projected/512e484b-c79d-4df3-bf9a-0502ed5bd338-kube-api-access-z8cg9\") pod \"payload-pre-processing-59fbc47bf8-t9h6w\" (UID: \"512e484b-c79d-4df3-bf9a-0502ed5bd338\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"
Sep 18 15:45:14.574286 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.574202    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8cg9\" (UniqueName: \"kubernetes.io/projected/512e484b-c79d-4df3-bf9a-0502ed5bd338-kube-api-access-z8cg9\") pod \"payload-pre-processing-59fbc47bf8-t9h6w\" (UID: \"512e484b-c79d-4df3-bf9a-0502ed5bd338\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"
Sep 18 15:45:14.574428 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.574289    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512e484b-c79d-4df3-bf9a-0502ed5bd338-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-t9h6w\" (UID: \"512e484b-c79d-4df3-bf9a-0502ed5bd338\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"
Sep 18 15:45:14.575204 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.575181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512e484b-c79d-4df3-bf9a-0502ed5bd338-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-t9h6w\" (UID: \"512e484b-c79d-4df3-bf9a-0502ed5bd338\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"
Sep 18 15:45:14.591343 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.591316    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8cg9\" (UniqueName: \"kubernetes.io/projected/512e484b-c79d-4df3-bf9a-0502ed5bd338-kube-api-access-z8cg9\") pod \"payload-pre-processing-59fbc47bf8-t9h6w\" (UID: \"512e484b-c79d-4df3-bf9a-0502ed5bd338\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"
Sep 18 15:45:14.630313 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.630283    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"
Sep 18 15:45:14.903536 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:14.903458    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"]
Sep 18 15:45:14.907068 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:45:14.907038    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512e484b_c79d_4df3_bf9a_0502ed5bd338.slice/crio-0aaf057911d447b018da9625b143eeda4c63c693f44f8885b8485a8c43fc7980 WatchSource:0}: Error finding container 0aaf057911d447b018da9625b143eeda4c63c693f44f8885b8485a8c43fc7980: Status 404 returned error can't find the container with id 0aaf057911d447b018da9625b143eeda4c63c693f44f8885b8485a8c43fc7980
Sep 18 15:45:15.371827 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:15.371784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w" event={"ID":"512e484b-c79d-4df3-bf9a-0502ed5bd338","Type":"ContainerStarted","Data":"0aaf057911d447b018da9625b143eeda4c63c693f44f8885b8485a8c43fc7980"}
Sep 18 15:45:16.379328 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:16.379298    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w" event={"ID":"512e484b-c79d-4df3-bf9a-0502ed5bd338","Type":"ContainerStarted","Data":"7773cca84e9f7578f5f6d9bf031c49ac6cb618fe3eab0d937acf3873f32d8b6c"}
Sep 18 15:45:16.379695 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:16.379351    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"
Sep 18 15:45:16.398137 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:16.398080    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w" podStartSLOduration=1.074027958 podStartE2EDuration="2.398063343s" totalImagesPullingTime="1.324035385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:45:16.396526158 +0000 UTC m=+659.941816833" watchObservedRunningTime="2026-09-18 15:45:16.398063343 +0000 UTC m=+659.943354017"
Sep 18 15:45:27.386408 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:27.386377    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"
Sep 18 15:45:44.475425 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.475383    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"]
Sep 18 15:45:44.479908 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.479884    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.483103 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.483061    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 15:45:44.484001 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.483980    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 15:45:44.484130 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.484003    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7lm9q\""
Sep 18 15:45:44.484211 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.484078    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 18 15:45:44.489946 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.489926    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"]
Sep 18 15:45:44.526041 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.526005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66b4807-817f-4732-9a5d-0f1bfd9b758b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.526210 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.526045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66b4807-817f-4732-9a5d-0f1bfd9b758b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.526210 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.526116    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66b4807-817f-4732-9a5d-0f1bfd9b758b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.526210 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.526193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2vn6\" (UniqueName: \"kubernetes.io/projected/c66b4807-817f-4732-9a5d-0f1bfd9b758b-kube-api-access-g2vn6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.526379 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.526241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66b4807-817f-4732-9a5d-0f1bfd9b758b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.526379 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.526290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66b4807-817f-4732-9a5d-0f1bfd9b758b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.627446 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.627419    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66b4807-817f-4732-9a5d-0f1bfd9b758b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.627585 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.627454    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66b4807-817f-4732-9a5d-0f1bfd9b758b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.627585 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.627486    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66b4807-817f-4732-9a5d-0f1bfd9b758b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.627585 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.627517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2vn6\" (UniqueName: \"kubernetes.io/projected/c66b4807-817f-4732-9a5d-0f1bfd9b758b-kube-api-access-g2vn6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.627585 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.627542    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66b4807-817f-4732-9a5d-0f1bfd9b758b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.627585 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.627575    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66b4807-817f-4732-9a5d-0f1bfd9b758b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.627906 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.627884    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c66b4807-817f-4732-9a5d-0f1bfd9b758b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.627964 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.627911    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c66b4807-817f-4732-9a5d-0f1bfd9b758b-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.627964 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.627953    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c66b4807-817f-4732-9a5d-0f1bfd9b758b-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.629740 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.629715    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c66b4807-817f-4732-9a5d-0f1bfd9b758b-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.630265 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.630245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c66b4807-817f-4732-9a5d-0f1bfd9b758b-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.636145 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.636127    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2vn6\" (UniqueName: \"kubernetes.io/projected/c66b4807-817f-4732-9a5d-0f1bfd9b758b-kube-api-access-g2vn6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dlxjl\" (UID: \"c66b4807-817f-4732-9a5d-0f1bfd9b758b\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:44.792813 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:44.792745    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:45.064090 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:45.064063    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"]
Sep 18 15:45:45.065706 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:45:45.065682    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66b4807_817f_4732_9a5d_0f1bfd9b758b.slice/crio-2f71b224b0d4d5f16766c53d3520c26743cdaae6f2355ee2e303b6da7c76e693 WatchSource:0}: Error finding container 2f71b224b0d4d5f16766c53d3520c26743cdaae6f2355ee2e303b6da7c76e693: Status 404 returned error can't find the container with id 2f71b224b0d4d5f16766c53d3520c26743cdaae6f2355ee2e303b6da7c76e693
Sep 18 15:45:45.067055 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:45.067037    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 15:45:45.521507 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:45.521463    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl" event={"ID":"c66b4807-817f-4732-9a5d-0f1bfd9b758b","Type":"ContainerStarted","Data":"2f71b224b0d4d5f16766c53d3520c26743cdaae6f2355ee2e303b6da7c76e693"}
Sep 18 15:45:46.527461 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:46.527420    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl" event={"ID":"c66b4807-817f-4732-9a5d-0f1bfd9b758b","Type":"ContainerStarted","Data":"91543bfb19b874c5337e7689b5556efa5627a17c2597bdddf2530498b8244b9d"}
Sep 18 15:45:46.546459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:46.546415    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl" podStartSLOduration=1.467044666 podStartE2EDuration="2.546400611s" totalImagesPullingTime="1.079355945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:45:46.545289057 +0000 UTC m=+690.090579734" watchObservedRunningTime="2026-09-18 15:45:46.546400611 +0000 UTC m=+690.091691305"
Sep 18 15:45:50.214816 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.214780    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"]
Sep 18 15:45:50.219354 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.219331    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.221910 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.221884    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 18 15:45:50.231949 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.231920    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"]
Sep 18 15:45:50.280721 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.280690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ed2659-68cc-4562-8e80-840d4a5dd71e-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.280721 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.280727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ed2659-68cc-4562-8e80-840d4a5dd71e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.280924 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.280786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ed2659-68cc-4562-8e80-840d4a5dd71e-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.280924 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.280807    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k259v\" (UniqueName: \"kubernetes.io/projected/65ed2659-68cc-4562-8e80-840d4a5dd71e-kube-api-access-k259v\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.280924 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.280893    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ed2659-68cc-4562-8e80-840d4a5dd71e-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.280924 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.280923    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ed2659-68cc-4562-8e80-840d4a5dd71e-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.381507 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.381476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ed2659-68cc-4562-8e80-840d4a5dd71e-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.381507 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.381511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ed2659-68cc-4562-8e80-840d4a5dd71e-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.381720 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.381556    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ed2659-68cc-4562-8e80-840d4a5dd71e-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.381720 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.381582    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ed2659-68cc-4562-8e80-840d4a5dd71e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.381720 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.381632    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ed2659-68cc-4562-8e80-840d4a5dd71e-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.381720 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.381682    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k259v\" (UniqueName: \"kubernetes.io/projected/65ed2659-68cc-4562-8e80-840d4a5dd71e-kube-api-access-k259v\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.381964 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.381944    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65ed2659-68cc-4562-8e80-840d4a5dd71e-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.382031 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.382009    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65ed2659-68cc-4562-8e80-840d4a5dd71e-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.382069 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.382035    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65ed2659-68cc-4562-8e80-840d4a5dd71e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.383914 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.383893    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65ed2659-68cc-4562-8e80-840d4a5dd71e-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.384114 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.384098    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65ed2659-68cc-4562-8e80-840d4a5dd71e-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.389472 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.389450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k259v\" (UniqueName: \"kubernetes.io/projected/65ed2659-68cc-4562-8e80-840d4a5dd71e-kube-api-access-k259v\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx\" (UID: \"65ed2659-68cc-4562-8e80-840d4a5dd71e\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.530755 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.530687    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:45:50.803702 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:50.803608    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"]
Sep 18 15:45:50.807392 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:45:50.807363    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ed2659_68cc_4562_8e80_840d4a5dd71e.slice/crio-e056fac3213b9c70c18d11af41a26539a4efeb8c289fb1dca6cc67c273fc7470 WatchSource:0}: Error finding container e056fac3213b9c70c18d11af41a26539a4efeb8c289fb1dca6cc67c273fc7470: Status 404 returned error can't find the container with id e056fac3213b9c70c18d11af41a26539a4efeb8c289fb1dca6cc67c273fc7470
Sep 18 15:45:51.562210 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:51.562169    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx" event={"ID":"65ed2659-68cc-4562-8e80-840d4a5dd71e","Type":"ContainerStarted","Data":"4c134002670e73132718b79059e606198bb3c46564f44eede1c6440c412cc39d"}
Sep 18 15:45:51.562698 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:51.562226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx" event={"ID":"65ed2659-68cc-4562-8e80-840d4a5dd71e","Type":"ContainerStarted","Data":"e056fac3213b9c70c18d11af41a26539a4efeb8c289fb1dca6cc67c273fc7470"}
Sep 18 15:45:51.583050 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:51.583005    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx" podStartSLOduration=1.399319909 podStartE2EDuration="1.58299004s" totalImagesPullingTime="183.670131ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:45:51.581914841 +0000 UTC m=+695.127205517" watchObservedRunningTime="2026-09-18 15:45:51.58299004 +0000 UTC m=+695.128280714"
Sep 18 15:45:54.793638 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:54.793603    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:54.794029 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:54.793683    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:54.806060 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:54.806035    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:45:55.591597 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:45:55.591565    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dlxjl"
Sep 18 15:46:00.531385 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:00.531349    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:46:00.531782 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:00.531421    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:46:00.544092 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:00.544053    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:46:00.621015 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:00.620981    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-z8rbx"
Sep 18 15:46:14.135109 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.135032    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"]
Sep 18 15:46:14.139213 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.139190    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.141903 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.141878    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 18 15:46:14.151197 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.151172    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"]
Sep 18 15:46:14.197656 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.197629    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/395c5de3-5da5-47ff-9ef5-9354b484ab03-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.197801 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.197686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/395c5de3-5da5-47ff-9ef5-9354b484ab03-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.197801 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.197704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/395c5de3-5da5-47ff-9ef5-9354b484ab03-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.197801 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.197752    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/395c5de3-5da5-47ff-9ef5-9354b484ab03-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.197905 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.197832    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw2z7\" (UniqueName: \"kubernetes.io/projected/395c5de3-5da5-47ff-9ef5-9354b484ab03-kube-api-access-sw2z7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.197905 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.197862    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395c5de3-5da5-47ff-9ef5-9354b484ab03-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.299282 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.299252    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw2z7\" (UniqueName: \"kubernetes.io/projected/395c5de3-5da5-47ff-9ef5-9354b484ab03-kube-api-access-sw2z7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.299395 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.299286    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395c5de3-5da5-47ff-9ef5-9354b484ab03-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.299395 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.299328    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/395c5de3-5da5-47ff-9ef5-9354b484ab03-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.299395 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.299355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/395c5de3-5da5-47ff-9ef5-9354b484ab03-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.299395 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.299370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/395c5de3-5da5-47ff-9ef5-9354b484ab03-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.299395 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.299384    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/395c5de3-5da5-47ff-9ef5-9354b484ab03-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.299771 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.299750    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/395c5de3-5da5-47ff-9ef5-9354b484ab03-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.299856 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.299829    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/395c5de3-5da5-47ff-9ef5-9354b484ab03-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.299956 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.299854    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/395c5de3-5da5-47ff-9ef5-9354b484ab03-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.301632 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.301614    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/395c5de3-5da5-47ff-9ef5-9354b484ab03-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.301805 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.301788    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/395c5de3-5da5-47ff-9ef5-9354b484ab03-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.308548 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.308524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw2z7\" (UniqueName: \"kubernetes.io/projected/395c5de3-5da5-47ff-9ef5-9354b484ab03-kube-api-access-sw2z7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm\" (UID: \"395c5de3-5da5-47ff-9ef5-9354b484ab03\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.452066 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.452038    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:14.722603 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:14.722582    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"]
Sep 18 15:46:14.725145 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:46:14.725115    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395c5de3_5da5_47ff_9ef5_9354b484ab03.slice/crio-d2ad56ed2ab30102d5e428e66b9e39e6c4925ae35c6ab48fc5beb3835b541e54 WatchSource:0}: Error finding container d2ad56ed2ab30102d5e428e66b9e39e6c4925ae35c6ab48fc5beb3835b541e54: Status 404 returned error can't find the container with id d2ad56ed2ab30102d5e428e66b9e39e6c4925ae35c6ab48fc5beb3835b541e54
Sep 18 15:46:15.676073 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:15.676035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm" event={"ID":"395c5de3-5da5-47ff-9ef5-9354b484ab03","Type":"ContainerStarted","Data":"e2e2d8f26097268f896908a0100633b5ce7bb6f595da3035dfe591f246386c5f"}
Sep 18 15:46:15.676447 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:15.676097    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm" event={"ID":"395c5de3-5da5-47ff-9ef5-9354b484ab03","Type":"ContainerStarted","Data":"d2ad56ed2ab30102d5e428e66b9e39e6c4925ae35c6ab48fc5beb3835b541e54"}
Sep 18 15:46:15.697095 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:15.697043    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm" podStartSLOduration=1.496027674 podStartE2EDuration="1.697023196s" totalImagesPullingTime="200.995522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:46:15.696863078 +0000 UTC m=+719.242153755" watchObservedRunningTime="2026-09-18 15:46:15.697023196 +0000 UTC m=+719.242313871"
Sep 18 15:46:24.452886 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:24.452852    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:24.453341 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:24.452930    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:24.465024 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:24.465002    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:24.733077 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:24.732980    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fkctsm"
Sep 18 15:46:31.014056 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.014023    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"]
Sep 18 15:46:31.018730 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.018712    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.021155 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.021138    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 18 15:46:31.027389 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.027369    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"]
Sep 18 15:46:31.144869 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.144841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f8f2849-7086-476c-aa88-171d8266ce4e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.145048 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.144907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8f2849-7086-476c-aa88-171d8266ce4e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.145048 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.144947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f8f2849-7086-476c-aa88-171d8266ce4e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.145048 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.144968    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kg65\" (UniqueName: \"kubernetes.io/projected/5f8f2849-7086-476c-aa88-171d8266ce4e-kube-api-access-4kg65\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.145204 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.145051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f8f2849-7086-476c-aa88-171d8266ce4e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.145204 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.145096    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f8f2849-7086-476c-aa88-171d8266ce4e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.246289 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.246253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f8f2849-7086-476c-aa88-171d8266ce4e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.246430 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.246338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8f2849-7086-476c-aa88-171d8266ce4e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.246430 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.246367    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f8f2849-7086-476c-aa88-171d8266ce4e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.246430 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.246397    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kg65\" (UniqueName: \"kubernetes.io/projected/5f8f2849-7086-476c-aa88-171d8266ce4e-kube-api-access-4kg65\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.246566 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.246460    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f8f2849-7086-476c-aa88-171d8266ce4e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.246566 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.246492    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f8f2849-7086-476c-aa88-171d8266ce4e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.246872 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.246846    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8f2849-7086-476c-aa88-171d8266ce4e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.246975 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.246894    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f8f2849-7086-476c-aa88-171d8266ce4e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.247069 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.247049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f8f2849-7086-476c-aa88-171d8266ce4e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.248617 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.248597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f8f2849-7086-476c-aa88-171d8266ce4e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.248935 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.248916    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f8f2849-7086-476c-aa88-171d8266ce4e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.255474 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.255456    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kg65\" (UniqueName: \"kubernetes.io/projected/5f8f2849-7086-476c-aa88-171d8266ce4e-kube-api-access-4kg65\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-l4d82\" (UID: \"5f8f2849-7086-476c-aa88-171d8266ce4e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.328079 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.328012    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:31.600936 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.600910    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"]
Sep 18 15:46:31.602450 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:46:31.602423    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f8f2849_7086_476c_aa88_171d8266ce4e.slice/crio-33c5bd4f396da41c64a7568413ad4c72c0ea8509221f5a3d543b3c3f34e0826f WatchSource:0}: Error finding container 33c5bd4f396da41c64a7568413ad4c72c0ea8509221f5a3d543b3c3f34e0826f: Status 404 returned error can't find the container with id 33c5bd4f396da41c64a7568413ad4c72c0ea8509221f5a3d543b3c3f34e0826f
Sep 18 15:46:31.755346 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:31.755318    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82" event={"ID":"5f8f2849-7086-476c-aa88-171d8266ce4e","Type":"ContainerStarted","Data":"33c5bd4f396da41c64a7568413ad4c72c0ea8509221f5a3d543b3c3f34e0826f"}
Sep 18 15:46:32.761584 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:32.761547    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82" event={"ID":"5f8f2849-7086-476c-aa88-171d8266ce4e","Type":"ContainerStarted","Data":"f25b0497b6bbdd656523e5200f704a520d8b438661ed02b3edfc3827466f8603"}
Sep 18 15:46:32.782983 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:32.782915    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82" podStartSLOduration=2.592452999 podStartE2EDuration="2.782900739s" totalImagesPullingTime="190.44774ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:46:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:46:32.781592901 +0000 UTC m=+736.326883566" watchObservedRunningTime="2026-09-18 15:46:32.782900739 +0000 UTC m=+736.328191464"
Sep 18 15:46:41.328345 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:41.328311    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:41.328791 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:41.328573    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:41.340699 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:41.340679    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:46:41.825103 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:46:41.825069    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-l4d82"
Sep 18 15:47:06.125909 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:06.125871    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d76dfb587-qlk8x"]
Sep 18 15:47:06.129937 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:06.129917    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d76dfb587-qlk8x"
Sep 18 15:47:06.139369 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:06.139344    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d76dfb587-qlk8x"]
Sep 18 15:47:06.152867 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:06.152845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6d9fb62-3526-42e7-8c2c-34a8b3eac395-tls-cert\") pod \"authorino-d76dfb587-qlk8x\" (UID: \"b6d9fb62-3526-42e7-8c2c-34a8b3eac395\") " pod="kuadrant-system/authorino-d76dfb587-qlk8x"
Sep 18 15:47:06.152995 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:06.152909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4lzk\" (UniqueName: \"kubernetes.io/projected/b6d9fb62-3526-42e7-8c2c-34a8b3eac395-kube-api-access-n4lzk\") pod \"authorino-d76dfb587-qlk8x\" (UID: \"b6d9fb62-3526-42e7-8c2c-34a8b3eac395\") " pod="kuadrant-system/authorino-d76dfb587-qlk8x"
Sep 18 15:47:06.253514 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:06.253477    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4lzk\" (UniqueName: \"kubernetes.io/projected/b6d9fb62-3526-42e7-8c2c-34a8b3eac395-kube-api-access-n4lzk\") pod \"authorino-d76dfb587-qlk8x\" (UID: \"b6d9fb62-3526-42e7-8c2c-34a8b3eac395\") " pod="kuadrant-system/authorino-d76dfb587-qlk8x"
Sep 18 15:47:06.253741 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:06.253577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6d9fb62-3526-42e7-8c2c-34a8b3eac395-tls-cert\") pod \"authorino-d76dfb587-qlk8x\" (UID: \"b6d9fb62-3526-42e7-8c2c-34a8b3eac395\") " pod="kuadrant-system/authorino-d76dfb587-qlk8x"
Sep 18 15:47:06.255992 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:06.255969    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b6d9fb62-3526-42e7-8c2c-34a8b3eac395-tls-cert\") pod \"authorino-d76dfb587-qlk8x\" (UID: \"b6d9fb62-3526-42e7-8c2c-34a8b3eac395\") " pod="kuadrant-system/authorino-d76dfb587-qlk8x"
Sep 18 15:47:06.268646 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:06.268620    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4lzk\" (UniqueName: \"kubernetes.io/projected/b6d9fb62-3526-42e7-8c2c-34a8b3eac395-kube-api-access-n4lzk\") pod \"authorino-d76dfb587-qlk8x\" (UID: \"b6d9fb62-3526-42e7-8c2c-34a8b3eac395\") " pod="kuadrant-system/authorino-d76dfb587-qlk8x"
Sep 18 15:47:06.440298 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:06.440265    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d76dfb587-qlk8x"
Sep 18 15:47:06.708924 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:06.708898    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d76dfb587-qlk8x"]
Sep 18 15:47:06.710570 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:47:06.710537    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6d9fb62_3526_42e7_8c2c_34a8b3eac395.slice/crio-63a367247a0bfdcb44603a44841f91faf43b085b0c80f7faf1cb27efcf6b10ed WatchSource:0}: Error finding container 63a367247a0bfdcb44603a44841f91faf43b085b0c80f7faf1cb27efcf6b10ed: Status 404 returned error can't find the container with id 63a367247a0bfdcb44603a44841f91faf43b085b0c80f7faf1cb27efcf6b10ed
Sep 18 15:47:06.937290 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:06.937261    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d76dfb587-qlk8x" event={"ID":"b6d9fb62-3526-42e7-8c2c-34a8b3eac395","Type":"ContainerStarted","Data":"63a367247a0bfdcb44603a44841f91faf43b085b0c80f7faf1cb27efcf6b10ed"}
Sep 18 15:47:07.945178 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:07.945139    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d76dfb587-qlk8x" event={"ID":"b6d9fb62-3526-42e7-8c2c-34a8b3eac395","Type":"ContainerStarted","Data":"764827b49c1c32269cf87c4e1d379c431c8ff5a6a13ddd4e8d5451ff0d31b795"}
Sep 18 15:47:07.973872 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:07.973804    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d76dfb587-qlk8x" podStartSLOduration=1.404823831 podStartE2EDuration="1.973780559s" totalImagesPullingTime="568.956728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:47:07.972009471 +0000 UTC m=+771.517300146" watchObservedRunningTime="2026-09-18 15:47:07.973780559 +0000 UTC m=+771.519071233"
Sep 18 15:47:08.016383 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.016348    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b7fcd96f8-mld8l"]
Sep 18 15:47:08.016605 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.016571    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b7fcd96f8-mld8l" podUID="ee4eae1a-0d12-4a97-bc66-652aa2f66c18" containerName="authorino" containerID="cri-o://95a65cdde10454f001bde5222d416c3c05d100db806cdfad74f84b319d38d9d5" gracePeriod=30
Sep 18 15:47:08.259914 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.259886    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b7fcd96f8-mld8l"
Sep 18 15:47:08.274732 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.274698    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s7m7\" (UniqueName: \"kubernetes.io/projected/ee4eae1a-0d12-4a97-bc66-652aa2f66c18-kube-api-access-9s7m7\") pod \"ee4eae1a-0d12-4a97-bc66-652aa2f66c18\" (UID: \"ee4eae1a-0d12-4a97-bc66-652aa2f66c18\") "
Sep 18 15:47:08.274893 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.274786    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee4eae1a-0d12-4a97-bc66-652aa2f66c18-tls-cert\") pod \"ee4eae1a-0d12-4a97-bc66-652aa2f66c18\" (UID: \"ee4eae1a-0d12-4a97-bc66-652aa2f66c18\") "
Sep 18 15:47:08.277001 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.276964    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4eae1a-0d12-4a97-bc66-652aa2f66c18-kube-api-access-9s7m7" (OuterVolumeSpecName: "kube-api-access-9s7m7") pod "ee4eae1a-0d12-4a97-bc66-652aa2f66c18" (UID: "ee4eae1a-0d12-4a97-bc66-652aa2f66c18"). InnerVolumeSpecName "kube-api-access-9s7m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:47:08.314713 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.314679    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee4eae1a-0d12-4a97-bc66-652aa2f66c18-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ee4eae1a-0d12-4a97-bc66-652aa2f66c18" (UID: "ee4eae1a-0d12-4a97-bc66-652aa2f66c18"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:47:08.376601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.376567    2561 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee4eae1a-0d12-4a97-bc66-652aa2f66c18-tls-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:47:08.376601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.376602    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s7m7\" (UniqueName: \"kubernetes.io/projected/ee4eae1a-0d12-4a97-bc66-652aa2f66c18-kube-api-access-9s7m7\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:47:08.636411 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.636328    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-zmld8"]
Sep 18 15:47:08.636558 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.636534    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8" podUID="c16a1987-43b8-4d38-b03f-f5da907d36ec" containerName="manager" containerID="cri-o://a11a7024185df5c97a83c86a5f33cd92c5167714d03b8b695cdbc43b488f46a1" gracePeriod=30
Sep 18 15:47:08.904466 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.904443    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:47:08.952274 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.952233    2561 generic.go:356] "Generic (PLEG): container finished" podID="ee4eae1a-0d12-4a97-bc66-652aa2f66c18" containerID="95a65cdde10454f001bde5222d416c3c05d100db806cdfad74f84b319d38d9d5" exitCode=0
Sep 18 15:47:08.952745 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.952276    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b7fcd96f8-mld8l"
Sep 18 15:47:08.952745 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.952311    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b7fcd96f8-mld8l" event={"ID":"ee4eae1a-0d12-4a97-bc66-652aa2f66c18","Type":"ContainerDied","Data":"95a65cdde10454f001bde5222d416c3c05d100db806cdfad74f84b319d38d9d5"}
Sep 18 15:47:08.952745 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.952446    2561 scope.go:117] "RemoveContainer" containerID="95a65cdde10454f001bde5222d416c3c05d100db806cdfad74f84b319d38d9d5"
Sep 18 15:47:08.952918 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.952355    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b7fcd96f8-mld8l" event={"ID":"ee4eae1a-0d12-4a97-bc66-652aa2f66c18","Type":"ContainerDied","Data":"221313a5b6924e79c7f3d7da65b5cdb627ef382c9083a144ac7a9c418eb6fe2a"}
Sep 18 15:47:08.954521 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.954500    2561 generic.go:356] "Generic (PLEG): container finished" podID="c16a1987-43b8-4d38-b03f-f5da907d36ec" containerID="a11a7024185df5c97a83c86a5f33cd92c5167714d03b8b695cdbc43b488f46a1" exitCode=0
Sep 18 15:47:08.955083 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.954642    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8" event={"ID":"c16a1987-43b8-4d38-b03f-f5da907d36ec","Type":"ContainerDied","Data":"a11a7024185df5c97a83c86a5f33cd92c5167714d03b8b695cdbc43b488f46a1"}
Sep 18 15:47:08.955083 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.954918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8" event={"ID":"c16a1987-43b8-4d38-b03f-f5da907d36ec","Type":"ContainerDied","Data":"679a96e3e403f9739d0753c4bf7864918b308f6cd4cd5a4333361cbcd98215fb"}
Sep 18 15:47:08.955083 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.954733    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-zmld8"
Sep 18 15:47:08.963999 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.963977    2561 scope.go:117] "RemoveContainer" containerID="95a65cdde10454f001bde5222d416c3c05d100db806cdfad74f84b319d38d9d5"
Sep 18 15:47:08.964291 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:47:08.964263    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a65cdde10454f001bde5222d416c3c05d100db806cdfad74f84b319d38d9d5\": container with ID starting with 95a65cdde10454f001bde5222d416c3c05d100db806cdfad74f84b319d38d9d5 not found: ID does not exist" containerID="95a65cdde10454f001bde5222d416c3c05d100db806cdfad74f84b319d38d9d5"
Sep 18 15:47:08.964372 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.964291    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a65cdde10454f001bde5222d416c3c05d100db806cdfad74f84b319d38d9d5"} err="failed to get container status \"95a65cdde10454f001bde5222d416c3c05d100db806cdfad74f84b319d38d9d5\": rpc error: code = NotFound desc = could not find container \"95a65cdde10454f001bde5222d416c3c05d100db806cdfad74f84b319d38d9d5\": container with ID starting with 95a65cdde10454f001bde5222d416c3c05d100db806cdfad74f84b319d38d9d5 not found: ID does not exist"
Sep 18 15:47:08.964372 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.964314    2561 scope.go:117] "RemoveContainer" containerID="a11a7024185df5c97a83c86a5f33cd92c5167714d03b8b695cdbc43b488f46a1"
Sep 18 15:47:08.973858 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.973838    2561 scope.go:117] "RemoveContainer" containerID="a11a7024185df5c97a83c86a5f33cd92c5167714d03b8b695cdbc43b488f46a1"
Sep 18 15:47:08.974226 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:47:08.974207    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11a7024185df5c97a83c86a5f33cd92c5167714d03b8b695cdbc43b488f46a1\": container with ID starting with a11a7024185df5c97a83c86a5f33cd92c5167714d03b8b695cdbc43b488f46a1 not found: ID does not exist" containerID="a11a7024185df5c97a83c86a5f33cd92c5167714d03b8b695cdbc43b488f46a1"
Sep 18 15:47:08.974312 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.974237    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11a7024185df5c97a83c86a5f33cd92c5167714d03b8b695cdbc43b488f46a1"} err="failed to get container status \"a11a7024185df5c97a83c86a5f33cd92c5167714d03b8b695cdbc43b488f46a1\": rpc error: code = NotFound desc = could not find container \"a11a7024185df5c97a83c86a5f33cd92c5167714d03b8b695cdbc43b488f46a1\": container with ID starting with a11a7024185df5c97a83c86a5f33cd92c5167714d03b8b695cdbc43b488f46a1 not found: ID does not exist"
Sep 18 15:47:08.980465 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.980180    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b7fcd96f8-mld8l"]
Sep 18 15:47:08.983037 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.983015    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16a1987-43b8-4d38-b03f-f5da907d36ec-metrics-certs\") pod \"c16a1987-43b8-4d38-b03f-f5da907d36ec\" (UID: \"c16a1987-43b8-4d38-b03f-f5da907d36ec\") "
Sep 18 15:47:08.983156 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.983080    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j7ss\" (UniqueName: \"kubernetes.io/projected/c16a1987-43b8-4d38-b03f-f5da907d36ec-kube-api-access-6j7ss\") pod \"c16a1987-43b8-4d38-b03f-f5da907d36ec\" (UID: \"c16a1987-43b8-4d38-b03f-f5da907d36ec\") "
Sep 18 15:47:08.983156 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.983114    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c16a1987-43b8-4d38-b03f-f5da907d36ec-webhook-cert\") pod \"c16a1987-43b8-4d38-b03f-f5da907d36ec\" (UID: \"c16a1987-43b8-4d38-b03f-f5da907d36ec\") "
Sep 18 15:47:08.983912 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.983879    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b7fcd96f8-mld8l"]
Sep 18 15:47:08.986892 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.986867    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16a1987-43b8-4d38-b03f-f5da907d36ec-kube-api-access-6j7ss" (OuterVolumeSpecName: "kube-api-access-6j7ss") pod "c16a1987-43b8-4d38-b03f-f5da907d36ec" (UID: "c16a1987-43b8-4d38-b03f-f5da907d36ec"). InnerVolumeSpecName "kube-api-access-6j7ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:47:08.988802 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.988769    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c16a1987-43b8-4d38-b03f-f5da907d36ec-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c16a1987-43b8-4d38-b03f-f5da907d36ec" (UID: "c16a1987-43b8-4d38-b03f-f5da907d36ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:47:08.989326 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.989302    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c16a1987-43b8-4d38-b03f-f5da907d36ec-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c16a1987-43b8-4d38-b03f-f5da907d36ec" (UID: "c16a1987-43b8-4d38-b03f-f5da907d36ec"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:47:08.995918 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.995888    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"]
Sep 18 15:47:08.996255 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:08.996115    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w" podUID="512e484b-c79d-4df3-bf9a-0502ed5bd338" containerName="payload-pre-processing" containerID="cri-o://7773cca84e9f7578f5f6d9bf031c49ac6cb618fe3eab0d937acf3873f32d8b6c" gracePeriod=30
Sep 18 15:47:09.084346 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.084314    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16a1987-43b8-4d38-b03f-f5da907d36ec-metrics-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:47:09.084346 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.084341    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j7ss\" (UniqueName: \"kubernetes.io/projected/c16a1987-43b8-4d38-b03f-f5da907d36ec-kube-api-access-6j7ss\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:47:09.084346 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.084351    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c16a1987-43b8-4d38-b03f-f5da907d36ec-webhook-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:47:09.242245 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.242218    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"
Sep 18 15:47:09.286449 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.286418    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512e484b-c79d-4df3-bf9a-0502ed5bd338-plugins-config-volume\") pod \"512e484b-c79d-4df3-bf9a-0502ed5bd338\" (UID: \"512e484b-c79d-4df3-bf9a-0502ed5bd338\") "
Sep 18 15:47:09.286625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.286554    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8cg9\" (UniqueName: \"kubernetes.io/projected/512e484b-c79d-4df3-bf9a-0502ed5bd338-kube-api-access-z8cg9\") pod \"512e484b-c79d-4df3-bf9a-0502ed5bd338\" (UID: \"512e484b-c79d-4df3-bf9a-0502ed5bd338\") "
Sep 18 15:47:09.286855 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.286831    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/512e484b-c79d-4df3-bf9a-0502ed5bd338-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "512e484b-c79d-4df3-bf9a-0502ed5bd338" (UID: "512e484b-c79d-4df3-bf9a-0502ed5bd338"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:47:09.288821 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.288793    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/512e484b-c79d-4df3-bf9a-0502ed5bd338-kube-api-access-z8cg9" (OuterVolumeSpecName: "kube-api-access-z8cg9") pod "512e484b-c79d-4df3-bf9a-0502ed5bd338" (UID: "512e484b-c79d-4df3-bf9a-0502ed5bd338"). InnerVolumeSpecName "kube-api-access-z8cg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:47:09.295530 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.295500    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-zmld8"]
Sep 18 15:47:09.312023 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.309968    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-zmld8"]
Sep 18 15:47:09.388044 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.388003    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/512e484b-c79d-4df3-bf9a-0502ed5bd338-plugins-config-volume\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:47:09.388044 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.388044    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8cg9\" (UniqueName: \"kubernetes.io/projected/512e484b-c79d-4df3-bf9a-0502ed5bd338-kube-api-access-z8cg9\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:47:09.706444 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.706407    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-7fxd2"]
Sep 18 15:47:09.706874 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.706857    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="512e484b-c79d-4df3-bf9a-0502ed5bd338" containerName="payload-pre-processing"
Sep 18 15:47:09.706874 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.706874    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="512e484b-c79d-4df3-bf9a-0502ed5bd338" containerName="payload-pre-processing"
Sep 18 15:47:09.707038 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.706896    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c16a1987-43b8-4d38-b03f-f5da907d36ec" containerName="manager"
Sep 18 15:47:09.707038 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.706903    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c16a1987-43b8-4d38-b03f-f5da907d36ec" containerName="manager"
Sep 18 15:47:09.707038 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.706921    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4eae1a-0d12-4a97-bc66-652aa2f66c18" containerName="authorino"
Sep 18 15:47:09.707038 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.706929    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4eae1a-0d12-4a97-bc66-652aa2f66c18" containerName="authorino"
Sep 18 15:47:09.707228 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.707045    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c16a1987-43b8-4d38-b03f-f5da907d36ec" containerName="manager"
Sep 18 15:47:09.707228 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.707059    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="512e484b-c79d-4df3-bf9a-0502ed5bd338" containerName="payload-pre-processing"
Sep 18 15:47:09.707228 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.707070    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee4eae1a-0d12-4a97-bc66-652aa2f66c18" containerName="authorino"
Sep 18 15:47:09.710186 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.710160    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 15:47:09.712868 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.712846    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qjgkv\""
Sep 18 15:47:09.713612 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.713594    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 15:47:09.713915 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.713899    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 15:47:09.714104 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.714089    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 15:47:09.723477 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.723457    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-7fxd2"]
Sep 18 15:47:09.792040 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.792005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzn62\" (UniqueName: \"kubernetes.io/projected/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-kube-api-access-gzn62\") pod \"maas-controller-7f9bc5dc6f-7fxd2\" (UID: \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\") " pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 15:47:09.792200 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.792049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-metrics-certs\") pod \"maas-controller-7f9bc5dc6f-7fxd2\" (UID: \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\") " pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 15:47:09.792200 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.792180    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-webhook-cert\") pod \"maas-controller-7f9bc5dc6f-7fxd2\" (UID: \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\") " pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 15:47:09.893572 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.893529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-metrics-certs\") pod \"maas-controller-7f9bc5dc6f-7fxd2\" (UID: \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\") " pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 15:47:09.893785 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.893654    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-webhook-cert\") pod \"maas-controller-7f9bc5dc6f-7fxd2\" (UID: \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\") " pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 15:47:09.893785 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.893774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzn62\" (UniqueName: \"kubernetes.io/projected/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-kube-api-access-gzn62\") pod \"maas-controller-7f9bc5dc6f-7fxd2\" (UID: \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\") " pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 15:47:09.896126 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.896094    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-webhook-cert\") pod \"maas-controller-7f9bc5dc6f-7fxd2\" (UID: \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\") " pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 15:47:09.896126 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.896118    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-metrics-certs\") pod \"maas-controller-7f9bc5dc6f-7fxd2\" (UID: \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\") " pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 15:47:09.902258 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.902233    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzn62\" (UniqueName: \"kubernetes.io/projected/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-kube-api-access-gzn62\") pod \"maas-controller-7f9bc5dc6f-7fxd2\" (UID: \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\") " pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 15:47:09.963173 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.963079    2561 generic.go:356] "Generic (PLEG): container finished" podID="512e484b-c79d-4df3-bf9a-0502ed5bd338" containerID="7773cca84e9f7578f5f6d9bf031c49ac6cb618fe3eab0d937acf3873f32d8b6c" exitCode=0
Sep 18 15:47:09.963173 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.963138    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"
Sep 18 15:47:09.963173 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.963157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w" event={"ID":"512e484b-c79d-4df3-bf9a-0502ed5bd338","Type":"ContainerDied","Data":"7773cca84e9f7578f5f6d9bf031c49ac6cb618fe3eab0d937acf3873f32d8b6c"}
Sep 18 15:47:09.963748 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.963211    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w" event={"ID":"512e484b-c79d-4df3-bf9a-0502ed5bd338","Type":"ContainerDied","Data":"0aaf057911d447b018da9625b143eeda4c63c693f44f8885b8485a8c43fc7980"}
Sep 18 15:47:09.963748 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.963235    2561 scope.go:117] "RemoveContainer" containerID="7773cca84e9f7578f5f6d9bf031c49ac6cb618fe3eab0d937acf3873f32d8b6c"
Sep 18 15:47:09.975567 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.975512    2561 scope.go:117] "RemoveContainer" containerID="7773cca84e9f7578f5f6d9bf031c49ac6cb618fe3eab0d937acf3873f32d8b6c"
Sep 18 15:47:09.976215 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:47:09.976193    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7773cca84e9f7578f5f6d9bf031c49ac6cb618fe3eab0d937acf3873f32d8b6c\": container with ID starting with 7773cca84e9f7578f5f6d9bf031c49ac6cb618fe3eab0d937acf3873f32d8b6c not found: ID does not exist" containerID="7773cca84e9f7578f5f6d9bf031c49ac6cb618fe3eab0d937acf3873f32d8b6c"
Sep 18 15:47:09.976308 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.976221    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7773cca84e9f7578f5f6d9bf031c49ac6cb618fe3eab0d937acf3873f32d8b6c"} err="failed to get container status \"7773cca84e9f7578f5f6d9bf031c49ac6cb618fe3eab0d937acf3873f32d8b6c\": rpc error: code = NotFound desc = could not find container \"7773cca84e9f7578f5f6d9bf031c49ac6cb618fe3eab0d937acf3873f32d8b6c\": container with ID starting with 7773cca84e9f7578f5f6d9bf031c49ac6cb618fe3eab0d937acf3873f32d8b6c not found: ID does not exist"
Sep 18 15:47:09.986630 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.986603    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"]
Sep 18 15:47:09.992472 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:09.992452    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-t9h6w"]
Sep 18 15:47:10.021601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:10.021554    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 15:47:10.294323 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:10.294301    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-7fxd2"]
Sep 18 15:47:10.296376 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:47:10.296342    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478ebc8d_edd3_4c32_8ca6_35a48d4707f8.slice/crio-f0a23490106b412faf59cf93db9e783136d238758ed4c619c0bec6ba06fd0087 WatchSource:0}: Error finding container f0a23490106b412faf59cf93db9e783136d238758ed4c619c0bec6ba06fd0087: Status 404 returned error can't find the container with id f0a23490106b412faf59cf93db9e783136d238758ed4c619c0bec6ba06fd0087
Sep 18 15:47:10.380778 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:10.380748    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="512e484b-c79d-4df3-bf9a-0502ed5bd338" path="/var/lib/kubelet/pods/512e484b-c79d-4df3-bf9a-0502ed5bd338/volumes"
Sep 18 15:47:10.381132 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:10.381118    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c16a1987-43b8-4d38-b03f-f5da907d36ec" path="/var/lib/kubelet/pods/c16a1987-43b8-4d38-b03f-f5da907d36ec/volumes"
Sep 18 15:47:10.381437 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:10.381426    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee4eae1a-0d12-4a97-bc66-652aa2f66c18" path="/var/lib/kubelet/pods/ee4eae1a-0d12-4a97-bc66-652aa2f66c18/volumes"
Sep 18 15:47:10.970265 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:10.970160    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2" event={"ID":"478ebc8d-edd3-4c32-8ca6-35a48d4707f8","Type":"ContainerStarted","Data":"16ede73e6f5d06ef9a96e0ef0a13e81dea57dad010bb2a2c93d888cfc041e85a"}
Sep 18 15:47:10.970265 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:10.970218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2" event={"ID":"478ebc8d-edd3-4c32-8ca6-35a48d4707f8","Type":"ContainerStarted","Data":"f0a23490106b412faf59cf93db9e783136d238758ed4c619c0bec6ba06fd0087"}
Sep 18 15:47:10.970779 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:10.970262    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 15:47:10.991199 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:10.991137    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2" podStartSLOduration=1.618935622 podStartE2EDuration="1.991115579s" totalImagesPullingTime="372.179957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:47:10.989781313 +0000 UTC m=+774.535071999" watchObservedRunningTime="2026-09-18 15:47:10.991115579 +0000 UTC m=+774.536406254"
Sep 18 15:47:21.978387 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:47:21.978360    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 15:48:10.199738 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:10.199703    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9143-62267c"]
Sep 18 15:48:10.204109 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:10.204085    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:10.207695 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:10.207645    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-w9zf6\""
Sep 18 15:48:10.210051 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:10.210029    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9143-62267c"]
Sep 18 15:48:10.324534 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:10.324503    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzmmd\" (UniqueName: \"kubernetes.io/projected/f6cf6663-241a-4625-9a85-328071cad4da-kube-api-access-mzmmd\") pod \"test-curl-9143-62267c\" (UID: \"f6cf6663-241a-4625-9a85-328071cad4da\") " pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:10.425679 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:10.425640    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzmmd\" (UniqueName: \"kubernetes.io/projected/f6cf6663-241a-4625-9a85-328071cad4da-kube-api-access-mzmmd\") pod \"test-curl-9143-62267c\" (UID: \"f6cf6663-241a-4625-9a85-328071cad4da\") " pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:10.435216 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:10.435189    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzmmd\" (UniqueName: \"kubernetes.io/projected/f6cf6663-241a-4625-9a85-328071cad4da-kube-api-access-mzmmd\") pod \"test-curl-9143-62267c\" (UID: \"f6cf6663-241a-4625-9a85-328071cad4da\") " pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:10.515340 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:10.515275    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:10.806363 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:10.806339    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9143-62267c"]
Sep 18 15:48:10.808159 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:48:10.808129    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6cf6663_241a_4625_9a85_328071cad4da.slice/crio-d7a81f78e498b1780e5f0d61ff21b096da531bb8f61291f447ac492bd528f950 WatchSource:0}: Error finding container d7a81f78e498b1780e5f0d61ff21b096da531bb8f61291f447ac492bd528f950: Status 404 returned error can't find the container with id d7a81f78e498b1780e5f0d61ff21b096da531bb8f61291f447ac492bd528f950
Sep 18 15:48:11.279015 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:11.278983    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9143-62267c" event={"ID":"f6cf6663-241a-4625-9a85-328071cad4da","Type":"ContainerStarted","Data":"d7a81f78e498b1780e5f0d61ff21b096da531bb8f61291f447ac492bd528f950"}
Sep 18 15:48:14.297142 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.297105    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9143-62267c" event={"ID":"f6cf6663-241a-4625-9a85-328071cad4da","Type":"ContainerStarted","Data":"becbfcdd2f5b0bc0af40ee25d9aa710106caa102f1f5f36cd2993bcd7472ab2a"}
Sep 18 15:48:14.312833 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.312783    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9143-62267c" podStartSLOduration=1.791551112 podStartE2EDuration="4.312766938s" totalImagesPullingTime="2.521215826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:14.311565211 +0000 UTC m=+837.856855888" watchObservedRunningTime="2026-09-18 15:48:14.312766938 +0000 UTC m=+837.858057612"
Sep 18 15:48:14.550994 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.550923    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"]
Sep 18 15:48:14.554646 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.554622    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"
Sep 18 15:48:14.558046 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.558022    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-998130-dockercfg-qbl4z\""
Sep 18 15:48:14.558046 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.558038    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-998130\""
Sep 18 15:48:14.565932 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.565911    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"]
Sep 18 15:48:14.664709 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.664649    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdz5d\" (UniqueName: \"kubernetes.io/projected/0bc8396a-f6c2-4385-ac55-19e2c982772a-kube-api-access-pdz5d\") pod \"payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8\" (UID: \"0bc8396a-f6c2-4385-ac55-19e2c982772a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"
Sep 18 15:48:14.664861 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.664782    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc8396a-f6c2-4385-ac55-19e2c982772a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8\" (UID: \"0bc8396a-f6c2-4385-ac55-19e2c982772a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"
Sep 18 15:48:14.765680 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.765632    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc8396a-f6c2-4385-ac55-19e2c982772a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8\" (UID: \"0bc8396a-f6c2-4385-ac55-19e2c982772a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"
Sep 18 15:48:14.765852 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.765724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdz5d\" (UniqueName: \"kubernetes.io/projected/0bc8396a-f6c2-4385-ac55-19e2c982772a-kube-api-access-pdz5d\") pod \"payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8\" (UID: \"0bc8396a-f6c2-4385-ac55-19e2c982772a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"
Sep 18 15:48:14.766419 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.766398    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc8396a-f6c2-4385-ac55-19e2c982772a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8\" (UID: \"0bc8396a-f6c2-4385-ac55-19e2c982772a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"
Sep 18 15:48:14.778325 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.778298    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdz5d\" (UniqueName: \"kubernetes.io/projected/0bc8396a-f6c2-4385-ac55-19e2c982772a-kube-api-access-pdz5d\") pod \"payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8\" (UID: \"0bc8396a-f6c2-4385-ac55-19e2c982772a\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"
Sep 18 15:48:14.865896 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.865824    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"
Sep 18 15:48:14.932908 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.932877    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9143-62267c"]
Sep 18 15:48:14.937687 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:14.937646    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9143-62267c"]
Sep 18 15:48:15.075266 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.075237    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9143-2a426e"]
Sep 18 15:48:15.075704 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.075684    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6cf6663-241a-4625-9a85-328071cad4da" containerName="test-curl-9143-62267c"
Sep 18 15:48:15.075764 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.075706    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6cf6663-241a-4625-9a85-328071cad4da" containerName="test-curl-9143-62267c"
Sep 18 15:48:15.075830 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.075819    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6cf6663-241a-4625-9a85-328071cad4da" containerName="test-curl-9143-62267c"
Sep 18 15:48:15.079933 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.079918    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9143-2a426e"
Sep 18 15:48:15.086426 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.086405    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9143-2a426e"]
Sep 18 15:48:15.143788 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.143767    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"]
Sep 18 15:48:15.144988 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:48:15.144964    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc8396a_f6c2_4385_ac55_19e2c982772a.slice/crio-a8211e93475583b326aa50f727cd443d598858e6349b9ec38f7fcd073dfe2623 WatchSource:0}: Error finding container a8211e93475583b326aa50f727cd443d598858e6349b9ec38f7fcd073dfe2623: Status 404 returned error can't find the container with id a8211e93475583b326aa50f727cd443d598858e6349b9ec38f7fcd073dfe2623
Sep 18 15:48:15.168692 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.168646    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-754k9\" (UniqueName: \"kubernetes.io/projected/97dc78bf-d75f-49ca-bd5a-b150fb37c116-kube-api-access-754k9\") pod \"test-curl-9143-2a426e\" (UID: \"97dc78bf-d75f-49ca-bd5a-b150fb37c116\") " pod="opendatahub/test-curl-9143-2a426e"
Sep 18 15:48:15.269652 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.269617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754k9\" (UniqueName: \"kubernetes.io/projected/97dc78bf-d75f-49ca-bd5a-b150fb37c116-kube-api-access-754k9\") pod \"test-curl-9143-2a426e\" (UID: \"97dc78bf-d75f-49ca-bd5a-b150fb37c116\") " pod="opendatahub/test-curl-9143-2a426e"
Sep 18 15:48:15.280120 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.280092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-754k9\" (UniqueName: \"kubernetes.io/projected/97dc78bf-d75f-49ca-bd5a-b150fb37c116-kube-api-access-754k9\") pod \"test-curl-9143-2a426e\" (UID: \"97dc78bf-d75f-49ca-bd5a-b150fb37c116\") " pod="opendatahub/test-curl-9143-2a426e"
Sep 18 15:48:15.304246 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.304218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8" event={"ID":"0bc8396a-f6c2-4385-ac55-19e2c982772a","Type":"ContainerStarted","Data":"aec9978a80b919d6daa5aa4182c2e055c4147826b24e2aa158faef913c283f8e"}
Sep 18 15:48:15.304597 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.304274    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8" event={"ID":"0bc8396a-f6c2-4385-ac55-19e2c982772a","Type":"ContainerStarted","Data":"a8211e93475583b326aa50f727cd443d598858e6349b9ec38f7fcd073dfe2623"}
Sep 18 15:48:15.304597 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:15.304434    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9143-62267c" container="test-curl-9143-62267c"
Sep 18 15:48:15.339078 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.339032    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8" podStartSLOduration=1.339019417 podStartE2EDuration="1.339019417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:15.335109753 +0000 UTC m=+838.880400427" watchObservedRunningTime="2026-09-18 15:48:15.339019417 +0000 UTC m=+838.884310091"
Sep 18 15:48:15.391674 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.391640    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9143-2a426e"
Sep 18 15:48:15.667630 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:15.667601    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9143-2a426e"]
Sep 18 15:48:15.669095 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:48:15.669064    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97dc78bf_d75f_49ca_bd5a_b150fb37c116.slice/crio-36aac1a51da32920987b489f50c34ada61fb471b52f43408762362037f611923 WatchSource:0}: Error finding container 36aac1a51da32920987b489f50c34ada61fb471b52f43408762362037f611923: Status 404 returned error can't find the container with id 36aac1a51da32920987b489f50c34ada61fb471b52f43408762362037f611923
Sep 18 15:48:16.311863 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:16.311824    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9143-2a426e" event={"ID":"97dc78bf-d75f-49ca-bd5a-b150fb37c116","Type":"ContainerStarted","Data":"bdd485b571c3386888bc827c964edd82e1bd4b7107ad02ba0e9351de55466883"}
Sep 18 15:48:16.312327 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:16.311871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9143-2a426e" event={"ID":"97dc78bf-d75f-49ca-bd5a-b150fb37c116","Type":"ContainerStarted","Data":"36aac1a51da32920987b489f50c34ada61fb471b52f43408762362037f611923"}
Sep 18 15:48:16.312327 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:16.311916    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9143-62267c" podUID="f6cf6663-241a-4625-9a85-328071cad4da" containerName="test-curl-9143-62267c" containerID="cri-o://becbfcdd2f5b0bc0af40ee25d9aa710106caa102f1f5f36cd2993bcd7472ab2a" gracePeriod=2
Sep 18 15:48:16.312327 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:16.312171    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"
Sep 18 15:48:16.328037 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:16.327991    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9143-2a426e" podStartSLOduration=1.327958341 podStartE2EDuration="1.327958341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:16.326156008 +0000 UTC m=+839.871446685" watchObservedRunningTime="2026-09-18 15:48:16.327958341 +0000 UTC m=+839.873249015"
Sep 18 15:48:16.328195 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:16.328173    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9143-62267c\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="f6cf6663-241a-4625-9a85-328071cad4da" pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:16.330040 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:16.330011    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9143-62267c\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="f6cf6663-241a-4625-9a85-328071cad4da" pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:16.991357 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:16.991277    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9143-2a426e"]
Sep 18 15:48:16.999976 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:16.999950    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9143-2a426e"]
Sep 18 15:48:17.317205 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:17.317117    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9143-2a426e" container="test-curl-9143-2a426e"
Sep 18 15:48:18.322309 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.322265    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9143-2a426e" podUID="97dc78bf-d75f-49ca-bd5a-b150fb37c116" containerName="test-curl-9143-2a426e" containerID="cri-o://bdd485b571c3386888bc827c964edd82e1bd4b7107ad02ba0e9351de55466883" gracePeriod=2
Sep 18 15:48:18.324618 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:18.324574    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9143-2a426e\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="97dc78bf-d75f-49ca-bd5a-b150fb37c116" pod="opendatahub/test-curl-9143-2a426e"
Sep 18 15:48:18.388221 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:18.388180    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9143-2a426e\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="97dc78bf-d75f-49ca-bd5a-b150fb37c116" pod="opendatahub/test-curl-9143-2a426e"
Sep 18 15:48:18.523944 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.523914    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:18.527931 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:18.527891    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9143-62267c\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="f6cf6663-241a-4625-9a85-328071cad4da" pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:18.599429 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.599353    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzmmd\" (UniqueName: \"kubernetes.io/projected/f6cf6663-241a-4625-9a85-328071cad4da-kube-api-access-mzmmd\") pod \"f6cf6663-241a-4625-9a85-328071cad4da\" (UID: \"f6cf6663-241a-4625-9a85-328071cad4da\") "
Sep 18 15:48:18.601428 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.601402    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6cf6663-241a-4625-9a85-328071cad4da-kube-api-access-mzmmd" (OuterVolumeSpecName: "kube-api-access-mzmmd") pod "f6cf6663-241a-4625-9a85-328071cad4da" (UID: "f6cf6663-241a-4625-9a85-328071cad4da"). InnerVolumeSpecName "kube-api-access-mzmmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:48:18.672223 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.672191    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"]
Sep 18 15:48:18.672629 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.672616    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97dc78bf-d75f-49ca-bd5a-b150fb37c116" containerName="test-curl-9143-2a426e"
Sep 18 15:48:18.672702 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.672632    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="97dc78bf-d75f-49ca-bd5a-b150fb37c116" containerName="test-curl-9143-2a426e"
Sep 18 15:48:18.672745 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.672713    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="97dc78bf-d75f-49ca-bd5a-b150fb37c116" containerName="test-curl-9143-2a426e"
Sep 18 15:48:18.675769 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.675754    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"
Sep 18 15:48:18.678108 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:18.678068    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9143-62267c\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="f6cf6663-241a-4625-9a85-328071cad4da" pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:18.678367 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.678345    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-9792ff-dockercfg-2znwk\""
Sep 18 15:48:18.679073 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.679056    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-9792ff\""
Sep 18 15:48:18.693526 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.693501    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"]
Sep 18 15:48:18.700741 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.700716    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzmmd\" (UniqueName: \"kubernetes.io/projected/f6cf6663-241a-4625-9a85-328071cad4da-kube-api-access-mzmmd\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:48:18.768492 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.768459    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"]
Sep 18 15:48:18.772012 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.771992    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"
Sep 18 15:48:18.774407 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:18.774381    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9143-62267c\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="f6cf6663-241a-4625-9a85-328071cad4da" pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:18.783374 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.783348    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"]
Sep 18 15:48:18.801345 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.801316    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb58ac36-8738-41cc-9cfc-c24652ddacd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2\" (UID: \"cb58ac36-8738-41cc-9cfc-c24652ddacd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"
Sep 18 15:48:18.801526 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.801510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxczk\" (UniqueName: \"kubernetes.io/projected/cb58ac36-8738-41cc-9cfc-c24652ddacd2-kube-api-access-cxczk\") pod \"payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2\" (UID: \"cb58ac36-8738-41cc-9cfc-c24652ddacd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"
Sep 18 15:48:18.902937 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.902845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c7ffbc-8e83-494b-a6c8-eefc7761b2eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc\" (UID: \"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"
Sep 18 15:48:18.902937 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.902899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp7lp\" (UniqueName: \"kubernetes.io/projected/15c7ffbc-8e83-494b-a6c8-eefc7761b2eb-kube-api-access-zp7lp\") pod \"payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc\" (UID: \"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"
Sep 18 15:48:18.903123 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.902943    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxczk\" (UniqueName: \"kubernetes.io/projected/cb58ac36-8738-41cc-9cfc-c24652ddacd2-kube-api-access-cxczk\") pod \"payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2\" (UID: \"cb58ac36-8738-41cc-9cfc-c24652ddacd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"
Sep 18 15:48:18.903123 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.903024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb58ac36-8738-41cc-9cfc-c24652ddacd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2\" (UID: \"cb58ac36-8738-41cc-9cfc-c24652ddacd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"
Sep 18 15:48:18.903917 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.903892    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb58ac36-8738-41cc-9cfc-c24652ddacd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2\" (UID: \"cb58ac36-8738-41cc-9cfc-c24652ddacd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"
Sep 18 15:48:18.912128 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.912105    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxczk\" (UniqueName: \"kubernetes.io/projected/cb58ac36-8738-41cc-9cfc-c24652ddacd2-kube-api-access-cxczk\") pod \"payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2\" (UID: \"cb58ac36-8738-41cc-9cfc-c24652ddacd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"
Sep 18 15:48:18.985753 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:18.985715    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"
Sep 18 15:48:19.004069 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:19.004028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c7ffbc-8e83-494b-a6c8-eefc7761b2eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc\" (UID: \"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"
Sep 18 15:48:19.004069 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:19.004087    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp7lp\" (UniqueName: \"kubernetes.io/projected/15c7ffbc-8e83-494b-a6c8-eefc7761b2eb-kube-api-access-zp7lp\") pod \"payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc\" (UID: \"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"
Sep 18 15:48:19.004771 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:19.004750    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c7ffbc-8e83-494b-a6c8-eefc7761b2eb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc\" (UID: \"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"
Sep 18 15:48:19.019031 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:19.019010    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp7lp\" (UniqueName: \"kubernetes.io/projected/15c7ffbc-8e83-494b-a6c8-eefc7761b2eb-kube-api-access-zp7lp\") pod \"payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc\" (UID: \"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"
Sep 18 15:48:19.083371 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:19.083334    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"
Sep 18 15:48:19.264903 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:19.264880    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"]
Sep 18 15:48:19.266695 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:48:19.266636    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb58ac36_8738_41cc_9cfc_c24652ddacd2.slice/crio-d72194ceb2b4558f9b509a14a8dab2094896018f8da93543377eb268c9e7bd2f WatchSource:0}: Error finding container d72194ceb2b4558f9b509a14a8dab2094896018f8da93543377eb268c9e7bd2f: Status 404 returned error can't find the container with id d72194ceb2b4558f9b509a14a8dab2094896018f8da93543377eb268c9e7bd2f
Sep 18 15:48:19.330727 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:19.330697    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2" event={"ID":"cb58ac36-8738-41cc-9cfc-c24652ddacd2","Type":"ContainerStarted","Data":"d72194ceb2b4558f9b509a14a8dab2094896018f8da93543377eb268c9e7bd2f"}
Sep 18 15:48:19.332120 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:19.332090    2561 generic.go:356] "Generic (PLEG): container finished" podID="f6cf6663-241a-4625-9a85-328071cad4da" containerID="becbfcdd2f5b0bc0af40ee25d9aa710106caa102f1f5f36cd2993bcd7472ab2a" exitCode=137
Sep 18 15:48:19.332239 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:19.332133    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:19.332239 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:19.332144    2561 scope.go:117] "RemoveContainer" containerID="becbfcdd2f5b0bc0af40ee25d9aa710106caa102f1f5f36cd2993bcd7472ab2a"
Sep 18 15:48:19.335705 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:19.335655    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9143-62267c\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="f6cf6663-241a-4625-9a85-328071cad4da" pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:19.342526 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:19.342502    2561 scope.go:117] "RemoveContainer" containerID="becbfcdd2f5b0bc0af40ee25d9aa710106caa102f1f5f36cd2993bcd7472ab2a"
Sep 18 15:48:19.342846 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:19.342816    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"becbfcdd2f5b0bc0af40ee25d9aa710106caa102f1f5f36cd2993bcd7472ab2a\": container with ID starting with becbfcdd2f5b0bc0af40ee25d9aa710106caa102f1f5f36cd2993bcd7472ab2a not found: ID does not exist" containerID="becbfcdd2f5b0bc0af40ee25d9aa710106caa102f1f5f36cd2993bcd7472ab2a"
Sep 18 15:48:19.342944 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:19.342847    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"becbfcdd2f5b0bc0af40ee25d9aa710106caa102f1f5f36cd2993bcd7472ab2a"} err="failed to get container status \"becbfcdd2f5b0bc0af40ee25d9aa710106caa102f1f5f36cd2993bcd7472ab2a\": rpc error: code = NotFound desc = could not find container \"becbfcdd2f5b0bc0af40ee25d9aa710106caa102f1f5f36cd2993bcd7472ab2a\": container with ID starting with becbfcdd2f5b0bc0af40ee25d9aa710106caa102f1f5f36cd2993bcd7472ab2a not found: ID does not exist"
Sep 18 15:48:19.344724 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:19.344688    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9143-62267c\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="f6cf6663-241a-4625-9a85-328071cad4da" pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:19.364675 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:19.364637    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"]
Sep 18 15:48:19.366656 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:48:19.366634    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c7ffbc_8e83_494b_a6c8_eefc7761b2eb.slice/crio-834cf388cfc5e7c875c687c2d3ee295bd99eebe8f07fe6b2d319c85b6274efa6 WatchSource:0}: Error finding container 834cf388cfc5e7c875c687c2d3ee295bd99eebe8f07fe6b2d319c85b6274efa6: Status 404 returned error can't find the container with id 834cf388cfc5e7c875c687c2d3ee295bd99eebe8f07fe6b2d319c85b6274efa6
Sep 18 15:48:20.340164 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:20.340135    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2" event={"ID":"cb58ac36-8738-41cc-9cfc-c24652ddacd2","Type":"ContainerStarted","Data":"9b55beaac3377ef2a37f3dd567057a53ef9237661becff68cd837fe214ed57b3"}
Sep 18 15:48:20.340521 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:20.340192    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"
Sep 18 15:48:20.344610 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:20.344563    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9143-62267c\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="f6cf6663-241a-4625-9a85-328071cad4da" pod="opendatahub/test-curl-9143-62267c"
Sep 18 15:48:20.344756 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:20.344734    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc" event={"ID":"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb","Type":"ContainerStarted","Data":"717aaff634b7e511d37c6a7b10da109b73c0c036571d98f2f081b9ff2f9e4231"}
Sep 18 15:48:20.344824 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:20.344777    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc" event={"ID":"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb","Type":"ContainerStarted","Data":"834cf388cfc5e7c875c687c2d3ee295bd99eebe8f07fe6b2d319c85b6274efa6"}
Sep 18 15:48:20.344914 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:20.344892    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"
Sep 18 15:48:20.380285 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:20.380262    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6cf6663-241a-4625-9a85-328071cad4da" path="/var/lib/kubelet/pods/f6cf6663-241a-4625-9a85-328071cad4da/volumes"
Sep 18 15:48:20.384062 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:20.384015    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2" podStartSLOduration=2.384000668 podStartE2EDuration="2.384000668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:20.383050285 +0000 UTC m=+843.928340959" watchObservedRunningTime="2026-09-18 15:48:20.384000668 +0000 UTC m=+843.929291342"
Sep 18 15:48:20.408606 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:20.408546    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc" podStartSLOduration=2.408525714 podStartE2EDuration="2.408525714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:20.408168354 +0000 UTC m=+843.953459028" watchObservedRunningTime="2026-09-18 15:48:20.408525714 +0000 UTC m=+843.953816389"
Sep 18 15:48:20.505803 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:20.505777    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9143-2a426e"
Sep 18 15:48:20.621829 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:20.621745    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-754k9\" (UniqueName: \"kubernetes.io/projected/97dc78bf-d75f-49ca-bd5a-b150fb37c116-kube-api-access-754k9\") pod \"97dc78bf-d75f-49ca-bd5a-b150fb37c116\" (UID: \"97dc78bf-d75f-49ca-bd5a-b150fb37c116\") "
Sep 18 15:48:20.623785 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:20.623762    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dc78bf-d75f-49ca-bd5a-b150fb37c116-kube-api-access-754k9" (OuterVolumeSpecName: "kube-api-access-754k9") pod "97dc78bf-d75f-49ca-bd5a-b150fb37c116" (UID: "97dc78bf-d75f-49ca-bd5a-b150fb37c116"). InnerVolumeSpecName "kube-api-access-754k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:48:20.722548 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:20.722511    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-754k9\" (UniqueName: \"kubernetes.io/projected/97dc78bf-d75f-49ca-bd5a-b150fb37c116-kube-api-access-754k9\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:48:21.350443 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:21.350408    2561 generic.go:356] "Generic (PLEG): container finished" podID="97dc78bf-d75f-49ca-bd5a-b150fb37c116" containerID="bdd485b571c3386888bc827c964edd82e1bd4b7107ad02ba0e9351de55466883" exitCode=137
Sep 18 15:48:21.350932 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:21.350462    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9143-2a426e"
Sep 18 15:48:21.350932 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:21.350509    2561 scope.go:117] "RemoveContainer" containerID="bdd485b571c3386888bc827c964edd82e1bd4b7107ad02ba0e9351de55466883"
Sep 18 15:48:21.359281 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:21.359262    2561 scope.go:117] "RemoveContainer" containerID="bdd485b571c3386888bc827c964edd82e1bd4b7107ad02ba0e9351de55466883"
Sep 18 15:48:21.359556 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:21.359538    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd485b571c3386888bc827c964edd82e1bd4b7107ad02ba0e9351de55466883\": container with ID starting with bdd485b571c3386888bc827c964edd82e1bd4b7107ad02ba0e9351de55466883 not found: ID does not exist" containerID="bdd485b571c3386888bc827c964edd82e1bd4b7107ad02ba0e9351de55466883"
Sep 18 15:48:21.359603 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:21.359564    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd485b571c3386888bc827c964edd82e1bd4b7107ad02ba0e9351de55466883"} err="failed to get container status \"bdd485b571c3386888bc827c964edd82e1bd4b7107ad02ba0e9351de55466883\": rpc error: code = NotFound desc = could not find container \"bdd485b571c3386888bc827c964edd82e1bd4b7107ad02ba0e9351de55466883\": container with ID starting with bdd485b571c3386888bc827c964edd82e1bd4b7107ad02ba0e9351de55466883 not found: ID does not exist"
Sep 18 15:48:21.360995 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:21.360974    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9143-2a426e\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="97dc78bf-d75f-49ca-bd5a-b150fb37c116" pod="opendatahub/test-curl-9143-2a426e"
Sep 18 15:48:22.379765 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:22.379736    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97dc78bf-d75f-49ca-bd5a-b150fb37c116" path="/var/lib/kubelet/pods/97dc78bf-d75f-49ca-bd5a-b150fb37c116/volumes"
Sep 18 15:48:26.814550 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.814513    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"]
Sep 18 15:48:26.820342 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.820321    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:26.823623 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.823605    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-70c6df-openshift-default-dockercfg-7prp9\""
Sep 18 15:48:26.823880 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.823864    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 15:48:26.838946 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.838922    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"]
Sep 18 15:48:26.984761 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.984728    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-data\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:26.984922 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.984763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27a9871-2cec-4728-9670-bfe1c9caf140-istiod-ca-cert\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:26.984922 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.984827    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-envoy\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:26.984922 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.984852    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t798s\" (UniqueName: \"kubernetes.io/projected/c27a9871-2cec-4728-9670-bfe1c9caf140-kube-api-access-t798s\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:26.985078 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.984950    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-workload-certs\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:26.985078 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.985002    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-workload-socket\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:26.985078 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.985037    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-podinfo\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:26.985222 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.985083    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-token\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:26.985222 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:26.985101    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-credential-socket\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.086336 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.086250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-envoy\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.086336 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.086287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t798s\" (UniqueName: \"kubernetes.io/projected/c27a9871-2cec-4728-9670-bfe1c9caf140-kube-api-access-t798s\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.086336 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.086327    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-workload-certs\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.086603 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.086361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-workload-socket\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.086603 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.086380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-podinfo\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.086603 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.086408    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-token\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.086603 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.086426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-credential-socket\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.086603 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.086461    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-data\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.086603 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.086478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27a9871-2cec-4728-9670-bfe1c9caf140-istiod-ca-cert\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.086951 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.086825    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-workload-socket\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.086951 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.086861    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-data\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.086951 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.086927    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-credential-socket\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.087074 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.086978    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-workload-certs\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.087135 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.087113    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27a9871-2cec-4728-9670-bfe1c9caf140-istiod-ca-cert\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.088554 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.088532    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-envoy\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.089105 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.089086    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-podinfo\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.095495 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.095476    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-token\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.096926 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.096905    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t798s\" (UniqueName: \"kubernetes.io/projected/c27a9871-2cec-4728-9670-bfe1c9caf140-kube-api-access-t798s\") pod \"e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") " pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.130883 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.130859    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:27.318984 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.318947    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"
Sep 18 15:48:27.413604 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:27.413581    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"]
Sep 18 15:48:27.415743 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:48:27.415715    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27a9871_2cec_4728_9670_bfe1c9caf140.slice/crio-cbdff6a890d5a29471387ee3e7734c2541dc4090400997c6ee59b7eb53851e0e WatchSource:0}: Error finding container cbdff6a890d5a29471387ee3e7734c2541dc4090400997c6ee59b7eb53851e0e: Status 404 returned error can't find the container with id cbdff6a890d5a29471387ee3e7734c2541dc4090400997c6ee59b7eb53851e0e
Sep 18 15:48:28.391225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:28.391182    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" event={"ID":"c27a9871-2cec-4728-9670-bfe1c9caf140","Type":"ContainerStarted","Data":"cbdff6a890d5a29471387ee3e7734c2541dc4090400997c6ee59b7eb53851e0e"}
Sep 18 15:48:30.039550 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:30.039511    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:48:30.039923 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:30.039582    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:48:30.039923 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:30.039610    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:48:30.405417 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:30.405330    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" event={"ID":"c27a9871-2cec-4728-9670-bfe1c9caf140","Type":"ContainerStarted","Data":"66a5c91764c11a61f04521f58b48641f7a0789c4d06606b9b9f7cc72663f0d67"}
Sep 18 15:48:30.436515 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:30.436450    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" podStartSLOduration=1.815210951 podStartE2EDuration="4.436427815s" totalImagesPullingTime="2.621216864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:30.434487728 +0000 UTC m=+853.979778403" watchObservedRunningTime="2026-09-18 15:48:30.436427815 +0000 UTC m=+853.981718491"
Sep 18 15:48:31.131911 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:31.131877    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:31.137141 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:31.137117    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:31.353215 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:31.353183    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"
Sep 18 15:48:31.353385 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:31.353265    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"
Sep 18 15:48:31.412842 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:31.412764    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:31.413828 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:31.413805    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:35.939599 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:35.939565    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"]
Sep 18 15:48:35.944381 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:35.944362    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:35.946968 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:35.946948    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-70c6df-serving-cert\""
Sep 18 15:48:35.948030 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:35.948011    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 15:48:35.948030 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:35.948026    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hp5gf\""
Sep 18 15:48:35.957235 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:35.957193    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"]
Sep 18 15:48:35.972426 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:35.972401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m47qb\" (UniqueName: \"kubernetes.io/projected/046ca9a4-1ee1-43b9-8005-50f73d7f098e-kube-api-access-m47qb\") pod \"maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:35.972542 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:35.972516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-maas-api-tls\") pod \"maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:35.972542 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:35.972540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/046ca9a4-1ee1-43b9-8005-50f73d7f098e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:35.972616 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:35.972559    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-metrics-certs\") pod \"maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:36.074077 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.074045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-maas-api-tls\") pod \"maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:36.074233 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.074082    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/046ca9a4-1ee1-43b9-8005-50f73d7f098e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:36.074233 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.074108    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-metrics-certs\") pod \"maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:36.074233 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.074163    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m47qb\" (UniqueName: \"kubernetes.io/projected/046ca9a4-1ee1-43b9-8005-50f73d7f098e-kube-api-access-m47qb\") pod \"maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:36.074233 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:36.074208    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-serving-cert: secret "maas-api-e2e-inf-b-70c6df-serving-cert" not found
Sep 18 15:48:36.074385 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:48:36.074280    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-maas-api-tls podName:046ca9a4-1ee1-43b9-8005-50f73d7f098e nodeName:}" failed. No retries permitted until 2026-09-18 15:48:36.574259281 +0000 UTC m=+860.119549947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-maas-api-tls") pod "maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp" (UID: "046ca9a4-1ee1-43b9-8005-50f73d7f098e") : secret "maas-api-e2e-inf-b-70c6df-serving-cert" not found
Sep 18 15:48:36.074833 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.074816    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/046ca9a4-1ee1-43b9-8005-50f73d7f098e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:36.076575 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.076554    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-metrics-certs\") pod \"maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:36.086321 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.086297    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m47qb\" (UniqueName: \"kubernetes.io/projected/046ca9a4-1ee1-43b9-8005-50f73d7f098e-kube-api-access-m47qb\") pod \"maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:36.580983 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.580950    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-maas-api-tls\") pod \"maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:36.583510 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.583486    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-maas-api-tls\") pod \"maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:36.674263 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.674232    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"]
Sep 18 15:48:36.678139 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.678112    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"
Sep 18 15:48:36.680841 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.680818    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-70c6df\""
Sep 18 15:48:36.680936 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.680895    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-70c6df-dockercfg-7zcwl\""
Sep 18 15:48:36.695594 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.695573    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"]
Sep 18 15:48:36.783685 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.783635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xvpw\" (UniqueName: \"kubernetes.io/projected/a1aa359a-5700-460b-b56f-5e7cdecfef9a-kube-api-access-6xvpw\") pod \"payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd\" (UID: \"a1aa359a-5700-460b-b56f-5e7cdecfef9a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"
Sep 18 15:48:36.783859 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.783712    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1aa359a-5700-460b-b56f-5e7cdecfef9a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd\" (UID: \"a1aa359a-5700-460b-b56f-5e7cdecfef9a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"
Sep 18 15:48:36.856158 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.856068    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:36.884637 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.884611    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1aa359a-5700-460b-b56f-5e7cdecfef9a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd\" (UID: \"a1aa359a-5700-460b-b56f-5e7cdecfef9a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"
Sep 18 15:48:36.884761 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.884733    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xvpw\" (UniqueName: \"kubernetes.io/projected/a1aa359a-5700-460b-b56f-5e7cdecfef9a-kube-api-access-6xvpw\") pod \"payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd\" (UID: \"a1aa359a-5700-460b-b56f-5e7cdecfef9a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"
Sep 18 15:48:36.885582 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.885558    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1aa359a-5700-460b-b56f-5e7cdecfef9a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd\" (UID: \"a1aa359a-5700-460b-b56f-5e7cdecfef9a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"
Sep 18 15:48:36.896070 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.896017    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xvpw\" (UniqueName: \"kubernetes.io/projected/a1aa359a-5700-460b-b56f-5e7cdecfef9a-kube-api-access-6xvpw\") pod \"payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd\" (UID: \"a1aa359a-5700-460b-b56f-5e7cdecfef9a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"
Sep 18 15:48:36.990289 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:36.990255    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"
Sep 18 15:48:37.136486 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:37.136458    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"]
Sep 18 15:48:37.138007 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:48:37.137983    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046ca9a4_1ee1_43b9_8005_50f73d7f098e.slice/crio-55de5320fd813501849a81ac816b7e5bc959de97d0e40fe5f08060c0dd6f6ca8 WatchSource:0}: Error finding container 55de5320fd813501849a81ac816b7e5bc959de97d0e40fe5f08060c0dd6f6ca8: Status 404 returned error can't find the container with id 55de5320fd813501849a81ac816b7e5bc959de97d0e40fe5f08060c0dd6f6ca8
Sep 18 15:48:37.268827 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:37.268746    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"]
Sep 18 15:48:37.270293 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:48:37.270265    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1aa359a_5700_460b_b56f_5e7cdecfef9a.slice/crio-1977236587bc49b72d2b1ec5ed95ee8498c0343cc6083c3a69f5244ba283b405 WatchSource:0}: Error finding container 1977236587bc49b72d2b1ec5ed95ee8498c0343cc6083c3a69f5244ba283b405: Status 404 returned error can't find the container with id 1977236587bc49b72d2b1ec5ed95ee8498c0343cc6083c3a69f5244ba283b405
Sep 18 15:48:37.448866 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:37.448832    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd" event={"ID":"a1aa359a-5700-460b-b56f-5e7cdecfef9a","Type":"ContainerStarted","Data":"d116dbd5becfa2a57f856f07e6832d7850041672c0b923c14cefb45a9cf356f6"}
Sep 18 15:48:37.449074 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:37.448892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd" event={"ID":"a1aa359a-5700-460b-b56f-5e7cdecfef9a","Type":"ContainerStarted","Data":"1977236587bc49b72d2b1ec5ed95ee8498c0343cc6083c3a69f5244ba283b405"}
Sep 18 15:48:37.449074 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:37.448937    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"
Sep 18 15:48:37.450920 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:37.450883    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp" event={"ID":"046ca9a4-1ee1-43b9-8005-50f73d7f098e","Type":"ContainerStarted","Data":"55de5320fd813501849a81ac816b7e5bc959de97d0e40fe5f08060c0dd6f6ca8"}
Sep 18 15:48:37.471321 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:37.471265    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd" podStartSLOduration=1.471249982 podStartE2EDuration="1.471249982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:37.46806533 +0000 UTC m=+861.013356004" watchObservedRunningTime="2026-09-18 15:48:37.471249982 +0000 UTC m=+861.016540689"
Sep 18 15:48:39.464528 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:39.464492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp" event={"ID":"046ca9a4-1ee1-43b9-8005-50f73d7f098e","Type":"ContainerStarted","Data":"10408680a237107663de96a19d15c30746f9081ce5d074ffa5e1ee2bc38c605d"}
Sep 18 15:48:39.464955 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:39.464839    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:39.485615 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:39.485572    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp" podStartSLOduration=2.750096997 podStartE2EDuration="4.485557145s" totalImagesPullingTime="1.735460148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:48:39.482523863 +0000 UTC m=+863.027814537" watchObservedRunningTime="2026-09-18 15:48:39.485557145 +0000 UTC m=+863.030847820"
Sep 18 15:48:45.475768 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:45.475743    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:48:49.467427 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:49.467397    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"
Sep 18 15:48:54.560191 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:54.560160    2561 generic.go:356] "Generic (PLEG): container finished" podID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerID="66a5c91764c11a61f04521f58b48641f7a0789c4d06606b9b9f7cc72663f0d67" exitCode=0
Sep 18 15:48:54.560587 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:54.560240    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" event={"ID":"c27a9871-2cec-4728-9670-bfe1c9caf140","Type":"ContainerDied","Data":"66a5c91764c11a61f04521f58b48641f7a0789c4d06606b9b9f7cc72663f0d67"}
Sep 18 15:48:54.560715 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:54.560700    2561 scope.go:117] "RemoveContainer" containerID="66a5c91764c11a61f04521f58b48641f7a0789c4d06606b9b9f7cc72663f0d67"
Sep 18 15:48:54.561877 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:54.561851    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:48:54.561948 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:54.561917    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:48:54.561985 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:54.561945    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:48:55.568172 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:55.568110    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" event={"ID":"c27a9871-2cec-4728-9670-bfe1c9caf140","Type":"ContainerStarted","Data":"3c0c5e440779ae1f7e8015b4dbf1f9ef81bb66b08d36e5927b86f130b999f528"}
Sep 18 15:48:56.131850 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:56.131814    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:48:56.132060 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:56.132015    2561 patch_prober.go:28] interesting pod/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 18 15:48:56.132139 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:56.132069    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 18 15:48:57.131992 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:57.131958    2561 patch_prober.go:28] interesting pod/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 18 15:48:57.132386 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:57.132023    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 18 15:48:58.131555 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:58.131526    2561 patch_prober.go:28] interesting pod/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 18 15:48:58.131744 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:58.131580    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 18 15:48:59.131423 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:59.131377    2561 patch_prober.go:28] interesting pod/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 18 15:48:59.131873 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:48:59.131450    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 18 15:49:00.132291 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:00.132252    2561 patch_prober.go:28] interesting pod/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Sep 18 15:49:00.132707 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:00.132314    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Sep 18 15:49:01.135242 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:01.135213    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:49:01.135746 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:01.135628    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:49:01.136324 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:01.136308    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:49:04.034099 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.034071    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"]
Sep 18 15:49:04.043111 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.043083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.045996 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.045972    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-998130\"/\"openshift-service-ca.crt\""
Sep 18 15:49:04.046143 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.045974    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-998130\"/\"default-dockercfg-lv7jt\""
Sep 18 15:49:04.046943 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.046923    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-998130\"/\"kube-root-ca.crt\""
Sep 18 15:49:04.047047 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.047032    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-998130\"/\"simulator-w0-998130-kserve-self-signed-certs\""
Sep 18 15:49:04.051834 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.051814    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"]
Sep 18 15:49:04.081330 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.081304    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-dshm\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.081464 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.081343    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-model-cache\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.081464 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.081405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-home\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.081593 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.081489    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22dww\" (UniqueName: \"kubernetes.io/projected/b35a3266-1689-4273-b948-7ac4ab16c5d3-kube-api-access-22dww\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.081593 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.081541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-tmp-dir\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.081593 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.081565    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b35a3266-1689-4273-b948-7ac4ab16c5d3-tls-certs\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.182885 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.182855    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22dww\" (UniqueName: \"kubernetes.io/projected/b35a3266-1689-4273-b948-7ac4ab16c5d3-kube-api-access-22dww\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.183045 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.182910    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-tmp-dir\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.183045 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.182938    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b35a3266-1689-4273-b948-7ac4ab16c5d3-tls-certs\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.183045 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.183011    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-dshm\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.183194 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.183058    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-model-cache\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.183194 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.183100    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-home\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.183347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.183318    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-tmp-dir\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.183438 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.183423    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-model-cache\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.183536 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.183516    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-home\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.185399 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.185367    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-dshm\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.185688 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.185646    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b35a3266-1689-4273-b948-7ac4ab16c5d3-tls-certs\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.198985 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.198927    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22dww\" (UniqueName: \"kubernetes.io/projected/b35a3266-1689-4273-b948-7ac4ab16c5d3-kube-api-access-22dww\") pod \"simulator-w0-998130-kserve-869fc5b49f-lc54l\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") " pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.375608 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.375519    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:04.660130 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:04.660102    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"]
Sep 18 15:49:04.661945 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:49:04.661919    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb35a3266_1689_4273_b948_7ac4ab16c5d3.slice/crio-8bc32c004ea0fdd9c94a3273504f7fa576339b4a3e1a78432b061a960f69c6af WatchSource:0}: Error finding container 8bc32c004ea0fdd9c94a3273504f7fa576339b4a3e1a78432b061a960f69c6af: Status 404 returned error can't find the container with id 8bc32c004ea0fdd9c94a3273504f7fa576339b4a3e1a78432b061a960f69c6af
Sep 18 15:49:05.621946 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:05.621906    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l" event={"ID":"b35a3266-1689-4273-b948-7ac4ab16c5d3","Type":"ContainerStarted","Data":"61df23f76366a50e9269534d17397e2982194c305f471545cc25c8b914f8fdff"}
Sep 18 15:49:05.621946 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:05.621953    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l" event={"ID":"b35a3266-1689-4273-b948-7ac4ab16c5d3","Type":"ContainerStarted","Data":"8bc32c004ea0fdd9c94a3273504f7fa576339b4a3e1a78432b061a960f69c6af"}
Sep 18 15:49:05.642495 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:05.642451    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l" podStartSLOduration=1.642435623 podStartE2EDuration="1.642435623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:49:05.641428466 +0000 UTC m=+889.186719133" watchObservedRunningTime="2026-09-18 15:49:05.642435623 +0000 UTC m=+889.187726297"
Sep 18 15:49:14.381126 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:14.381088    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:14.381126 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:14.381133    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:14.389559 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:14.389538    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:14.680125 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:14.680102    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:49:18.351960 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:18.351922    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 15:49:18.353170 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:18.353147    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 15:49:45.155516 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.155485    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"]
Sep 18 15:49:45.158134 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.158117    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.160836 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.160815    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9792ff\"/\"default-dockercfg-6xggs\""
Sep 18 15:49:45.161890 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.161870    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9792ff\"/\"openshift-service-ca.crt\""
Sep 18 15:49:45.161989 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.161894    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-9792ff\"/\"premium-w3-9792ff-kserve-self-signed-certs\""
Sep 18 15:49:45.161989 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.161920    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-9792ff\"/\"kube-root-ca.crt\""
Sep 18 15:49:45.176049 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.176022    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"]
Sep 18 15:49:45.179516 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.179493    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk875\" (UniqueName: \"kubernetes.io/projected/0928eec3-5a76-414f-8858-2505f8bedbee-kube-api-access-tk875\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.179622 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.179528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-model-cache\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.179622 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.179560    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0928eec3-5a76-414f-8858-2505f8bedbee-tls-certs\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.179732 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.179620    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-tmp-dir\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.179732 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.179705    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-home\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.179823 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.179762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-dshm\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.280673 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.280623    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-tmp-dir\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.280941 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.280695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-home\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.280941 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.280750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-dshm\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.280941 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.280809    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk875\" (UniqueName: \"kubernetes.io/projected/0928eec3-5a76-414f-8858-2505f8bedbee-kube-api-access-tk875\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.280941 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.280827    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-model-cache\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.280941 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.280845    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0928eec3-5a76-414f-8858-2505f8bedbee-tls-certs\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.281221 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.281066    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-tmp-dir\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.281221 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.281100    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-home\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.281221 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.281181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-model-cache\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.283102 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.283080    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-dshm\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.283412 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.283393    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0928eec3-5a76-414f-8858-2505f8bedbee-tls-certs\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.289780 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.289758    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk875\" (UniqueName: \"kubernetes.io/projected/0928eec3-5a76-414f-8858-2505f8bedbee-kube-api-access-tk875\") pod \"premium-w3-9792ff-kserve-6c775758cb-jrjl7\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") " pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.467905 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.467864    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:45.743725 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.743700    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"]
Sep 18 15:49:45.745918 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:49:45.745888    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0928eec3_5a76_414f_8858_2505f8bedbee.slice/crio-8e63a5dc5a32b6e69fbe1791a946246bd33b95edeffe87849bacc85b983fdff7 WatchSource:0}: Error finding container 8e63a5dc5a32b6e69fbe1791a946246bd33b95edeffe87849bacc85b983fdff7: Status 404 returned error can't find the container with id 8e63a5dc5a32b6e69fbe1791a946246bd33b95edeffe87849bacc85b983fdff7
Sep 18 15:49:45.837255 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.837218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7" event={"ID":"0928eec3-5a76-414f-8858-2505f8bedbee","Type":"ContainerStarted","Data":"f0b1374635a9ce2474b398ca55c6abff81affe19a2b38c52338f3ed806e3d0c6"}
Sep 18 15:49:45.837416 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.837265    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7" event={"ID":"0928eec3-5a76-414f-8858-2505f8bedbee","Type":"ContainerStarted","Data":"8e63a5dc5a32b6e69fbe1791a946246bd33b95edeffe87849bacc85b983fdff7"}
Sep 18 15:49:45.874440 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:45.874383    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7" podStartSLOduration=0.874362157 podStartE2EDuration="874.362157ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:49:45.874175544 +0000 UTC m=+929.419466220" watchObservedRunningTime="2026-09-18 15:49:45.874362157 +0000 UTC m=+929.419652832"
Sep 18 15:49:55.468417 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:55.468377    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:55.468875 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:55.468430    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:55.482475 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:55.482448    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:49:55.902729 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:49:55.902634    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:52:05.416762 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.416634    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"]
Sep 18 15:52:05.420757 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.420734    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.423345 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.423322    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-998130\"/\"distinct-w0-998130-kserve-self-signed-certs\""
Sep 18 15:52:05.429938 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.429912    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"]
Sep 18 15:52:05.546439 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.546407    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-home\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.546439 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.546442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-dshm\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.546637 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.546470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-tmp-dir\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.546637 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.546501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-model-cache\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.546637 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.546519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjtmg\" (UniqueName: \"kubernetes.io/projected/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-kube-api-access-kjtmg\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.546871 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.546707    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-tls-certs\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.647822 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.647792    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-home\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.647822 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.647825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-dshm\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.648045 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.647850    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-tmp-dir\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.648045 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.647878    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-model-cache\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.648045 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.647905    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtmg\" (UniqueName: \"kubernetes.io/projected/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-kube-api-access-kjtmg\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.648045 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.648003    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-tls-certs\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.648387 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.648356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-home\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.648387 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.648382    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-tmp-dir\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.648530 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.648414    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-model-cache\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.650227 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.650201    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-dshm\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.650360 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.650345    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-tls-certs\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.656367 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.656351    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtmg\" (UniqueName: \"kubernetes.io/projected/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-kube-api-access-kjtmg\") pod \"distinct-w0-998130-kserve-c475bf765-hbkfv\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") " pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:05.731061 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:05.731031    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:06.003008 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:06.002946    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"]
Sep 18 15:52:06.005818 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:52:06.005784    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44cbf9bb_7c0f_4e98_81a5_569fac0bbcf4.slice/crio-bbab6cbf1a0ce59ec6f5a9efd5fc2978fa252df7226e9fdec5e79bb182df094b WatchSource:0}: Error finding container bbab6cbf1a0ce59ec6f5a9efd5fc2978fa252df7226e9fdec5e79bb182df094b: Status 404 returned error can't find the container with id bbab6cbf1a0ce59ec6f5a9efd5fc2978fa252df7226e9fdec5e79bb182df094b
Sep 18 15:52:06.580364 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:06.580329    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv" event={"ID":"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4","Type":"ContainerStarted","Data":"803b02d0a331f8646c025fd5770efd24976c06a450040a74780b327bb9642865"}
Sep 18 15:52:06.580364 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:06.580370    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv" event={"ID":"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4","Type":"ContainerStarted","Data":"bbab6cbf1a0ce59ec6f5a9efd5fc2978fa252df7226e9fdec5e79bb182df094b"}
Sep 18 15:52:06.601419 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:06.601370    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv" podStartSLOduration=1.601355415 podStartE2EDuration="1.601355415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:52:06.598905608 +0000 UTC m=+1070.144196281" watchObservedRunningTime="2026-09-18 15:52:06.601355415 +0000 UTC m=+1070.146646127"
Sep 18 15:52:15.731848 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:15.731815    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:15.732240 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:15.731869    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:15.744600 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:15.744577    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:16.643064 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:16.643037    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:52:49.940957 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:49.940922    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"]
Sep 18 15:52:49.941530 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:49.941185    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp" podUID="046ca9a4-1ee1-43b9-8005-50f73d7f098e" containerName="maas-api" containerID="cri-o://10408680a237107663de96a19d15c30746f9081ce5d074ffa5e1ee2bc38c605d" gracePeriod=30
Sep 18 15:52:50.000288 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.000256    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"]
Sep 18 15:52:50.000552 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.000503    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd" podUID="a1aa359a-5700-460b-b56f-5e7cdecfef9a" containerName="payload-processing" containerID="cri-o://d116dbd5becfa2a57f856f07e6832d7850041672c0b923c14cefb45a9cf356f6" gracePeriod=30
Sep 18 15:52:50.250989 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.250967    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:52:50.280599 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.279794    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/046ca9a4-1ee1-43b9-8005-50f73d7f098e-openshift-service-ca\") pod \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") "
Sep 18 15:52:50.280599 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.279921    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-metrics-certs\") pod \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") "
Sep 18 15:52:50.280599 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.279951    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-maas-api-tls\") pod \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") "
Sep 18 15:52:50.280599 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.280061    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m47qb\" (UniqueName: \"kubernetes.io/projected/046ca9a4-1ee1-43b9-8005-50f73d7f098e-kube-api-access-m47qb\") pod \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\" (UID: \"046ca9a4-1ee1-43b9-8005-50f73d7f098e\") "
Sep 18 15:52:50.281808 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.280995    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/046ca9a4-1ee1-43b9-8005-50f73d7f098e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "046ca9a4-1ee1-43b9-8005-50f73d7f098e" (UID: "046ca9a4-1ee1-43b9-8005-50f73d7f098e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:52:50.283230 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.283203    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "046ca9a4-1ee1-43b9-8005-50f73d7f098e" (UID: "046ca9a4-1ee1-43b9-8005-50f73d7f098e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:52:50.283532 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.283482    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/046ca9a4-1ee1-43b9-8005-50f73d7f098e-kube-api-access-m47qb" (OuterVolumeSpecName: "kube-api-access-m47qb") pod "046ca9a4-1ee1-43b9-8005-50f73d7f098e" (UID: "046ca9a4-1ee1-43b9-8005-50f73d7f098e"). InnerVolumeSpecName "kube-api-access-m47qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:52:50.284808 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.284788    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "046ca9a4-1ee1-43b9-8005-50f73d7f098e" (UID: "046ca9a4-1ee1-43b9-8005-50f73d7f098e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:52:50.286023 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.286001    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"
Sep 18 15:52:50.380719 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.380688    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xvpw\" (UniqueName: \"kubernetes.io/projected/a1aa359a-5700-460b-b56f-5e7cdecfef9a-kube-api-access-6xvpw\") pod \"a1aa359a-5700-460b-b56f-5e7cdecfef9a\" (UID: \"a1aa359a-5700-460b-b56f-5e7cdecfef9a\") "
Sep 18 15:52:50.380914 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.380899    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1aa359a-5700-460b-b56f-5e7cdecfef9a-plugins-config-volume\") pod \"a1aa359a-5700-460b-b56f-5e7cdecfef9a\" (UID: \"a1aa359a-5700-460b-b56f-5e7cdecfef9a\") "
Sep 18 15:52:50.381190 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.381169    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-metrics-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:52:50.381238 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.381196    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/046ca9a4-1ee1-43b9-8005-50f73d7f098e-maas-api-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:52:50.381238 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.381210    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m47qb\" (UniqueName: \"kubernetes.io/projected/046ca9a4-1ee1-43b9-8005-50f73d7f098e-kube-api-access-m47qb\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:52:50.381238 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.381225    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/046ca9a4-1ee1-43b9-8005-50f73d7f098e-openshift-service-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:52:50.381337 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.381254    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1aa359a-5700-460b-b56f-5e7cdecfef9a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1aa359a-5700-460b-b56f-5e7cdecfef9a" (UID: "a1aa359a-5700-460b-b56f-5e7cdecfef9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:52:50.382766 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.382742    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1aa359a-5700-460b-b56f-5e7cdecfef9a-kube-api-access-6xvpw" (OuterVolumeSpecName: "kube-api-access-6xvpw") pod "a1aa359a-5700-460b-b56f-5e7cdecfef9a" (UID: "a1aa359a-5700-460b-b56f-5e7cdecfef9a"). InnerVolumeSpecName "kube-api-access-6xvpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:52:50.482064 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.481996    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1aa359a-5700-460b-b56f-5e7cdecfef9a-plugins-config-volume\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:52:50.482064 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.482019    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xvpw\" (UniqueName: \"kubernetes.io/projected/a1aa359a-5700-460b-b56f-5e7cdecfef9a-kube-api-access-6xvpw\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:52:50.811493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.811406    2561 generic.go:356] "Generic (PLEG): container finished" podID="a1aa359a-5700-460b-b56f-5e7cdecfef9a" containerID="d116dbd5becfa2a57f856f07e6832d7850041672c0b923c14cefb45a9cf356f6" exitCode=0
Sep 18 15:52:50.811493 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.811455    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"
Sep 18 15:52:50.811773 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.811492    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd" event={"ID":"a1aa359a-5700-460b-b56f-5e7cdecfef9a","Type":"ContainerDied","Data":"d116dbd5becfa2a57f856f07e6832d7850041672c0b923c14cefb45a9cf356f6"}
Sep 18 15:52:50.811773 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.811553    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd" event={"ID":"a1aa359a-5700-460b-b56f-5e7cdecfef9a","Type":"ContainerDied","Data":"1977236587bc49b72d2b1ec5ed95ee8498c0343cc6083c3a69f5244ba283b405"}
Sep 18 15:52:50.811773 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.811585    2561 scope.go:117] "RemoveContainer" containerID="d116dbd5becfa2a57f856f07e6832d7850041672c0b923c14cefb45a9cf356f6"
Sep 18 15:52:50.812946 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.812928    2561 generic.go:356] "Generic (PLEG): container finished" podID="046ca9a4-1ee1-43b9-8005-50f73d7f098e" containerID="10408680a237107663de96a19d15c30746f9081ce5d074ffa5e1ee2bc38c605d" exitCode=0
Sep 18 15:52:50.813036 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.812969    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"
Sep 18 15:52:50.813036 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.812997    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp" event={"ID":"046ca9a4-1ee1-43b9-8005-50f73d7f098e","Type":"ContainerDied","Data":"10408680a237107663de96a19d15c30746f9081ce5d074ffa5e1ee2bc38c605d"}
Sep 18 15:52:50.813156 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.813035    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp" event={"ID":"046ca9a4-1ee1-43b9-8005-50f73d7f098e","Type":"ContainerDied","Data":"55de5320fd813501849a81ac816b7e5bc959de97d0e40fe5f08060c0dd6f6ca8"}
Sep 18 15:52:50.821178 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.821162    2561 scope.go:117] "RemoveContainer" containerID="d116dbd5becfa2a57f856f07e6832d7850041672c0b923c14cefb45a9cf356f6"
Sep 18 15:52:50.821435 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:52:50.821417    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d116dbd5becfa2a57f856f07e6832d7850041672c0b923c14cefb45a9cf356f6\": container with ID starting with d116dbd5becfa2a57f856f07e6832d7850041672c0b923c14cefb45a9cf356f6 not found: ID does not exist" containerID="d116dbd5becfa2a57f856f07e6832d7850041672c0b923c14cefb45a9cf356f6"
Sep 18 15:52:50.821506 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.821446    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d116dbd5becfa2a57f856f07e6832d7850041672c0b923c14cefb45a9cf356f6"} err="failed to get container status \"d116dbd5becfa2a57f856f07e6832d7850041672c0b923c14cefb45a9cf356f6\": rpc error: code = NotFound desc = could not find container \"d116dbd5becfa2a57f856f07e6832d7850041672c0b923c14cefb45a9cf356f6\": container with ID starting with d116dbd5becfa2a57f856f07e6832d7850041672c0b923c14cefb45a9cf356f6 not found: ID does not exist"
Sep 18 15:52:50.821506 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.821468    2561 scope.go:117] "RemoveContainer" containerID="10408680a237107663de96a19d15c30746f9081ce5d074ffa5e1ee2bc38c605d"
Sep 18 15:52:50.830397 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.830379    2561 scope.go:117] "RemoveContainer" containerID="10408680a237107663de96a19d15c30746f9081ce5d074ffa5e1ee2bc38c605d"
Sep 18 15:52:50.830819 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:52:50.830793    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10408680a237107663de96a19d15c30746f9081ce5d074ffa5e1ee2bc38c605d\": container with ID starting with 10408680a237107663de96a19d15c30746f9081ce5d074ffa5e1ee2bc38c605d not found: ID does not exist" containerID="10408680a237107663de96a19d15c30746f9081ce5d074ffa5e1ee2bc38c605d"
Sep 18 15:52:50.831049 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.830942    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10408680a237107663de96a19d15c30746f9081ce5d074ffa5e1ee2bc38c605d"} err="failed to get container status \"10408680a237107663de96a19d15c30746f9081ce5d074ffa5e1ee2bc38c605d\": rpc error: code = NotFound desc = could not find container \"10408680a237107663de96a19d15c30746f9081ce5d074ffa5e1ee2bc38c605d\": container with ID starting with 10408680a237107663de96a19d15c30746f9081ce5d074ffa5e1ee2bc38c605d not found: ID does not exist"
Sep 18 15:52:50.833253 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.833233    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"]
Sep 18 15:52:50.837746 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.837727    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-70c6df-6674678b6d-7rcgp"]
Sep 18 15:52:50.848040 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.848021    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"]
Sep 18 15:52:50.854598 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:50.854576    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-70c6df-7ffffb4787-krmjd"]
Sep 18 15:52:52.380503 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:52.380464    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="046ca9a4-1ee1-43b9-8005-50f73d7f098e" path="/var/lib/kubelet/pods/046ca9a4-1ee1-43b9-8005-50f73d7f098e/volumes"
Sep 18 15:52:52.381061 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:52.381038    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1aa359a-5700-460b-b56f-5e7cdecfef9a" path="/var/lib/kubelet/pods/a1aa359a-5700-460b-b56f-5e7cdecfef9a/volumes"
Sep 18 15:52:57.078255 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:57.078200    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"]
Sep 18 15:52:57.078734 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:52:57.078482    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerName="istio-proxy" containerID="cri-o://3c0c5e440779ae1f7e8015b4dbf1f9ef81bb66b08d36e5927b86f130b999f528" gracePeriod=30
Sep 18 15:53:01.137882 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:01.137846    2561 patch_prober.go:28] interesting pod/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 15:53:01.138281 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:01.137904    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 15:53:02.340228 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.340161    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:53:02.399258 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.399231    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-workload-certs\") pod \"c27a9871-2cec-4728-9670-bfe1c9caf140\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") "
Sep 18 15:53:02.399258 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.399261    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-credential-socket\") pod \"c27a9871-2cec-4728-9670-bfe1c9caf140\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") "
Sep 18 15:53:02.399508 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.399292    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-data\") pod \"c27a9871-2cec-4728-9670-bfe1c9caf140\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") "
Sep 18 15:53:02.399508 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.399317    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-podinfo\") pod \"c27a9871-2cec-4728-9670-bfe1c9caf140\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") "
Sep 18 15:53:02.399508 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.399394    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-token\") pod \"c27a9871-2cec-4728-9670-bfe1c9caf140\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") "
Sep 18 15:53:02.399508 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.399429    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-workload-socket\") pod \"c27a9871-2cec-4728-9670-bfe1c9caf140\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") "
Sep 18 15:53:02.399508 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.399499    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27a9871-2cec-4728-9670-bfe1c9caf140-istiod-ca-cert\") pod \"c27a9871-2cec-4728-9670-bfe1c9caf140\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") "
Sep 18 15:53:02.399797 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.399522    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t798s\" (UniqueName: \"kubernetes.io/projected/c27a9871-2cec-4728-9670-bfe1c9caf140-kube-api-access-t798s\") pod \"c27a9871-2cec-4728-9670-bfe1c9caf140\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") "
Sep 18 15:53:02.399797 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.399586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c27a9871-2cec-4728-9670-bfe1c9caf140" (UID: "c27a9871-2cec-4728-9670-bfe1c9caf140"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:53:02.399797 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.399605    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-data" (OuterVolumeSpecName: "istio-data") pod "c27a9871-2cec-4728-9670-bfe1c9caf140" (UID: "c27a9871-2cec-4728-9670-bfe1c9caf140"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:53:02.399797 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.399702    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-envoy\") pod \"c27a9871-2cec-4728-9670-bfe1c9caf140\" (UID: \"c27a9871-2cec-4728-9670-bfe1c9caf140\") "
Sep 18 15:53:02.400017 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.399819    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c27a9871-2cec-4728-9670-bfe1c9caf140" (UID: "c27a9871-2cec-4728-9670-bfe1c9caf140"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:53:02.400017 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.399971    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27a9871-2cec-4728-9670-bfe1c9caf140-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c27a9871-2cec-4728-9670-bfe1c9caf140" (UID: "c27a9871-2cec-4728-9670-bfe1c9caf140"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:53:02.400126 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.400091    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c27a9871-2cec-4728-9670-bfe1c9caf140-istiod-ca-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:53:02.400126 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.400108    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-credential-socket\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:53:02.400126 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.400122    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-data\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:53:02.400280 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.400135    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-workload-socket\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:53:02.400280 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.400202    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c27a9871-2cec-4728-9670-bfe1c9caf140" (UID: "c27a9871-2cec-4728-9670-bfe1c9caf140"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:53:02.401848 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.401808    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27a9871-2cec-4728-9670-bfe1c9caf140-kube-api-access-t798s" (OuterVolumeSpecName: "kube-api-access-t798s") pod "c27a9871-2cec-4728-9670-bfe1c9caf140" (UID: "c27a9871-2cec-4728-9670-bfe1c9caf140"). InnerVolumeSpecName "kube-api-access-t798s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:53:02.401973 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.401897    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c27a9871-2cec-4728-9670-bfe1c9caf140" (UID: "c27a9871-2cec-4728-9670-bfe1c9caf140"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:53:02.402111 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.402086    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-token" (OuterVolumeSpecName: "istio-token") pod "c27a9871-2cec-4728-9670-bfe1c9caf140" (UID: "c27a9871-2cec-4728-9670-bfe1c9caf140"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:53:02.402111 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.402112    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c27a9871-2cec-4728-9670-bfe1c9caf140" (UID: "c27a9871-2cec-4728-9670-bfe1c9caf140"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 15:53:02.501425 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.501395    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-workload-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:53:02.501425 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.501420    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-podinfo\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:53:02.501425 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.501429    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-token\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:53:02.501699 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.501438    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t798s\" (UniqueName: \"kubernetes.io/projected/c27a9871-2cec-4728-9670-bfe1c9caf140-kube-api-access-t798s\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:53:02.501699 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.501447    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c27a9871-2cec-4728-9670-bfe1c9caf140-istio-envoy\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:53:02.884926 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.884886    2561 generic.go:356] "Generic (PLEG): container finished" podID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerID="3c0c5e440779ae1f7e8015b4dbf1f9ef81bb66b08d36e5927b86f130b999f528" exitCode=0
Sep 18 15:53:02.885094 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.884960    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"
Sep 18 15:53:02.885094 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.884969    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" event={"ID":"c27a9871-2cec-4728-9670-bfe1c9caf140","Type":"ContainerDied","Data":"3c0c5e440779ae1f7e8015b4dbf1f9ef81bb66b08d36e5927b86f130b999f528"}
Sep 18 15:53:02.885094 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.885061    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67" event={"ID":"c27a9871-2cec-4728-9670-bfe1c9caf140","Type":"ContainerDied","Data":"cbdff6a890d5a29471387ee3e7734c2541dc4090400997c6ee59b7eb53851e0e"}
Sep 18 15:53:02.885094 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.885092    2561 scope.go:117] "RemoveContainer" containerID="3c0c5e440779ae1f7e8015b4dbf1f9ef81bb66b08d36e5927b86f130b999f528"
Sep 18 15:53:02.895164 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.895147    2561 scope.go:117] "RemoveContainer" containerID="66a5c91764c11a61f04521f58b48641f7a0789c4d06606b9b9f7cc72663f0d67"
Sep 18 15:53:02.903968 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.903952    2561 scope.go:117] "RemoveContainer" containerID="3c0c5e440779ae1f7e8015b4dbf1f9ef81bb66b08d36e5927b86f130b999f528"
Sep 18 15:53:02.904216 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:53:02.904200    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0c5e440779ae1f7e8015b4dbf1f9ef81bb66b08d36e5927b86f130b999f528\": container with ID starting with 3c0c5e440779ae1f7e8015b4dbf1f9ef81bb66b08d36e5927b86f130b999f528 not found: ID does not exist" containerID="3c0c5e440779ae1f7e8015b4dbf1f9ef81bb66b08d36e5927b86f130b999f528"
Sep 18 15:53:02.904263 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.904222    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0c5e440779ae1f7e8015b4dbf1f9ef81bb66b08d36e5927b86f130b999f528"} err="failed to get container status \"3c0c5e440779ae1f7e8015b4dbf1f9ef81bb66b08d36e5927b86f130b999f528\": rpc error: code = NotFound desc = could not find container \"3c0c5e440779ae1f7e8015b4dbf1f9ef81bb66b08d36e5927b86f130b999f528\": container with ID starting with 3c0c5e440779ae1f7e8015b4dbf1f9ef81bb66b08d36e5927b86f130b999f528 not found: ID does not exist"
Sep 18 15:53:02.904263 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.904237    2561 scope.go:117] "RemoveContainer" containerID="66a5c91764c11a61f04521f58b48641f7a0789c4d06606b9b9f7cc72663f0d67"
Sep 18 15:53:02.904491 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:53:02.904474    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66a5c91764c11a61f04521f58b48641f7a0789c4d06606b9b9f7cc72663f0d67\": container with ID starting with 66a5c91764c11a61f04521f58b48641f7a0789c4d06606b9b9f7cc72663f0d67 not found: ID does not exist" containerID="66a5c91764c11a61f04521f58b48641f7a0789c4d06606b9b9f7cc72663f0d67"
Sep 18 15:53:02.904532 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.904497    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66a5c91764c11a61f04521f58b48641f7a0789c4d06606b9b9f7cc72663f0d67"} err="failed to get container status \"66a5c91764c11a61f04521f58b48641f7a0789c4d06606b9b9f7cc72663f0d67\": rpc error: code = NotFound desc = could not find container \"66a5c91764c11a61f04521f58b48641f7a0789c4d06606b9b9f7cc72663f0d67\": container with ID starting with 66a5c91764c11a61f04521f58b48641f7a0789c4d06606b9b9f7cc72663f0d67 not found: ID does not exist"
Sep 18 15:53:02.908359 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.908339    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"]
Sep 18 15:53:02.913478 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:02.913434    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-70c6df-openshift-default-6d5594fc8d-5vh67"]
Sep 18 15:53:04.380136 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.380100    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" path="/var/lib/kubelet/pods/c27a9871-2cec-4728-9670-bfe1c9caf140/volumes"
Sep 18 15:53:04.506763 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.506723    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"]
Sep 18 15:53:04.507213 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.507195    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerName="istio-proxy"
Sep 18 15:53:04.507317 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.507216    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerName="istio-proxy"
Sep 18 15:53:04.507317 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.507235    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerName="istio-proxy"
Sep 18 15:53:04.507317 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.507242    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerName="istio-proxy"
Sep 18 15:53:04.507317 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.507272    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1aa359a-5700-460b-b56f-5e7cdecfef9a" containerName="payload-processing"
Sep 18 15:53:04.507317 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.507282    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1aa359a-5700-460b-b56f-5e7cdecfef9a" containerName="payload-processing"
Sep 18 15:53:04.507317 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.507292    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="046ca9a4-1ee1-43b9-8005-50f73d7f098e" containerName="maas-api"
Sep 18 15:53:04.507317 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.507300    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="046ca9a4-1ee1-43b9-8005-50f73d7f098e" containerName="maas-api"
Sep 18 15:53:04.507652 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.507412    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="046ca9a4-1ee1-43b9-8005-50f73d7f098e" containerName="maas-api"
Sep 18 15:53:04.507652 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.507423    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerName="istio-proxy"
Sep 18 15:53:04.507652 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.507434    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27a9871-2cec-4728-9670-bfe1c9caf140" containerName="istio-proxy"
Sep 18 15:53:04.507652 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.507446    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1aa359a-5700-460b-b56f-5e7cdecfef9a" containerName="payload-processing"
Sep 18 15:53:04.512240 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.512221    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:04.515313 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.515292    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-f14aba-serving-cert\""
Sep 18 15:53:04.515831 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.515811    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hp5gf\""
Sep 18 15:53:04.515943 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.515865    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 15:53:04.537558 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.537529    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"]
Sep 18 15:53:04.623655 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.623625    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:04.623870 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.623706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f093215b-e864-4090-9c21-1760112dfaa8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:04.623870 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.623778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:04.623870 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.623834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqmhk\" (UniqueName: \"kubernetes.io/projected/f093215b-e864-4090-9c21-1760112dfaa8-kube-api-access-dqmhk\") pod \"maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:04.724983 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.724956    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:04.725147 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.724989    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f093215b-e864-4090-9c21-1760112dfaa8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:04.725147 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.725024    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:04.725147 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.725080    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqmhk\" (UniqueName: \"kubernetes.io/projected/f093215b-e864-4090-9c21-1760112dfaa8-kube-api-access-dqmhk\") pod \"maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:04.725282 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:53:04.725201    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-serving-cert: secret "maas-api-e2e-shared-a-w2-f14aba-serving-cert" not found
Sep 18 15:53:04.725331 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:53:04.725283    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-maas-api-tls podName:f093215b-e864-4090-9c21-1760112dfaa8 nodeName:}" failed. No retries permitted until 2026-09-18 15:53:05.225264022 +0000 UTC m=+1128.770554680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-maas-api-tls") pod "maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm" (UID: "f093215b-e864-4090-9c21-1760112dfaa8") : secret "maas-api-e2e-shared-a-w2-f14aba-serving-cert" not found
Sep 18 15:53:04.725700 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.725656    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f093215b-e864-4090-9c21-1760112dfaa8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:04.727507 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.727481    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:04.736775 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:04.736750    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqmhk\" (UniqueName: \"kubernetes.io/projected/f093215b-e864-4090-9c21-1760112dfaa8-kube-api-access-dqmhk\") pod \"maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:05.230151 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:05.230119    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:05.232597 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:05.232571    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:05.423917 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:05.423881    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:05.703999 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:05.703971    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"]
Sep 18 15:53:05.705182 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:53:05.705158    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf093215b_e864_4090_9c21_1760112dfaa8.slice/crio-2d5904380d74bfcb04e491e2f6587d9e2eb822d3f6ca3ee35c2fbb8cb0ba6936 WatchSource:0}: Error finding container 2d5904380d74bfcb04e491e2f6587d9e2eb822d3f6ca3ee35c2fbb8cb0ba6936: Status 404 returned error can't find the container with id 2d5904380d74bfcb04e491e2f6587d9e2eb822d3f6ca3ee35c2fbb8cb0ba6936
Sep 18 15:53:05.706567 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:05.706546    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 15:53:05.904080 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:05.904001    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm" event={"ID":"f093215b-e864-4090-9c21-1760112dfaa8","Type":"ContainerStarted","Data":"2d5904380d74bfcb04e491e2f6587d9e2eb822d3f6ca3ee35c2fbb8cb0ba6936"}
Sep 18 15:53:06.911054 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:06.911021    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm" event={"ID":"f093215b-e864-4090-9c21-1760112dfaa8","Type":"ContainerStarted","Data":"6c99600f928aeee4533897cb90f8c052a1f89837f9cda59a7ccdbc9fb1872dc4"}
Sep 18 15:53:06.911431 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:06.911146    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:06.932944 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:06.932896    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm" podStartSLOduration=2.512509844 podStartE2EDuration="2.932882225s" totalImagesPullingTime="420.372381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:53:06.930631661 +0000 UTC m=+1130.475922336" watchObservedRunningTime="2026-09-18 15:53:06.932882225 +0000 UTC m=+1130.478172934"
Sep 18 15:53:12.922149 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:12.922120    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 15:53:50.651637 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.651553    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"]
Sep 18 15:53:50.655623 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.655592    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:50.658735 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.658699    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-195881-serving-cert\""
Sep 18 15:53:50.667308 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.667286    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"]
Sep 18 15:53:50.766598 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.766567    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htgqd\" (UniqueName: \"kubernetes.io/projected/a261d245-7fea-4a2c-9cda-05f141288650-kube-api-access-htgqd\") pod \"maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:50.766786 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.766706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:50.766786 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.766744    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a261d245-7fea-4a2c-9cda-05f141288650-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:50.766786 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.766774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:50.868126 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.868088    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a261d245-7fea-4a2c-9cda-05f141288650-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:50.868126 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.868135    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:50.868358 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.868172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htgqd\" (UniqueName: \"kubernetes.io/projected/a261d245-7fea-4a2c-9cda-05f141288650-kube-api-access-htgqd\") pod \"maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:50.868358 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:53:50.868308    2561 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-serving-cert: secret "maas-api-e2e-shared-a-w1-195881-serving-cert" not found
Sep 18 15:53:50.868358 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.868341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:50.868511 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:53:50.868385    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-maas-api-tls podName:a261d245-7fea-4a2c-9cda-05f141288650 nodeName:}" failed. No retries permitted until 2026-09-18 15:53:51.368364127 +0000 UTC m=+1174.913654798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-maas-api-tls") pod "maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr" (UID: "a261d245-7fea-4a2c-9cda-05f141288650") : secret "maas-api-e2e-shared-a-w1-195881-serving-cert" not found
Sep 18 15:53:50.868902 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.868874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a261d245-7fea-4a2c-9cda-05f141288650-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:50.870704 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.870687    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:50.877161 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:50.877141    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htgqd\" (UniqueName: \"kubernetes.io/projected/a261d245-7fea-4a2c-9cda-05f141288650-kube-api-access-htgqd\") pod \"maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:51.374062 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:51.374034    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:51.376485 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:51.376461    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:51.566972 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:51.566939    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:51.839040 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:51.839015    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"]
Sep 18 15:53:51.841185 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:53:51.841160    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda261d245_7fea_4a2c_9cda_05f141288650.slice/crio-55f865c5b36a2511095d6e5b17cc20360aa00afc600d7a9d3932c92a1a829d9b WatchSource:0}: Error finding container 55f865c5b36a2511095d6e5b17cc20360aa00afc600d7a9d3932c92a1a829d9b: Status 404 returned error can't find the container with id 55f865c5b36a2511095d6e5b17cc20360aa00afc600d7a9d3932c92a1a829d9b
Sep 18 15:53:52.155937 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:52.155849    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr" event={"ID":"a261d245-7fea-4a2c-9cda-05f141288650","Type":"ContainerStarted","Data":"55f865c5b36a2511095d6e5b17cc20360aa00afc600d7a9d3932c92a1a829d9b"}
Sep 18 15:53:53.163626 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:53.163590    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr" event={"ID":"a261d245-7fea-4a2c-9cda-05f141288650","Type":"ContainerStarted","Data":"c3c13945bff3962deceba89ac2c856fb78804b337961c836402e7e2a1486ed57"}
Sep 18 15:53:53.164046 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:53.163768    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:53:53.194683 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:53.188110    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr" podStartSLOduration=2.811780263 podStartE2EDuration="3.188089052s" totalImagesPullingTime="376.308789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:53:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:53:53.185466617 +0000 UTC m=+1176.730757290" watchObservedRunningTime="2026-09-18 15:53:53.188089052 +0000 UTC m=+1176.733379794"
Sep 18 15:53:59.174193 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:53:59.174159    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 15:54:17.275026 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.274995    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"]
Sep 18 15:54:17.278786 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.278770    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.282877 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.282856    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 15:54:17.283636 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.283609    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-655d9c-openshift-default-dockercfg-bwtxl\""
Sep 18 15:54:17.297154 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.297123    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"]
Sep 18 15:54:17.318867 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.318843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istiod-ca-cert\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.318990 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.318874    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq8jg\" (UniqueName: \"kubernetes.io/projected/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-kube-api-access-dq8jg\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.318990 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.318899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-envoy\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.318990 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.318981    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-workload-certs\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.319102 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.319015    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-podinfo\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.319102 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.319030    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-token\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.319102 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.319063    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-data\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.319199 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.319147    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-workload-socket\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.319199 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.319165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-credential-socket\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.420322 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.420287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-workload-certs\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.420499 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.420334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-podinfo\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.420499 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.420370    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-token\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.420499 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.420411    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-data\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.420760 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.420502    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-workload-socket\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.420760 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.420527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-credential-socket\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.420760 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.420617    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istiod-ca-cert\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.420760 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.420705    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq8jg\" (UniqueName: \"kubernetes.io/projected/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-kube-api-access-dq8jg\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.420760 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.420752    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-workload-certs\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.421018 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.420772    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-envoy\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.421018 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.420853    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-data\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.421018 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.420919    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-credential-socket\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.421018 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.420955    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-workload-socket\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.421303 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.421285    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istiod-ca-cert\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.423008 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.422988    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-envoy\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.423094 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.423061    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-podinfo\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.430090 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.430069    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-token\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.430208 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.430154    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq8jg\" (UniqueName: \"kubernetes.io/projected/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-kube-api-access-dq8jg\") pod \"e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") " pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.592325 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.592240    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:17.866770 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.866743    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"]
Sep 18 15:54:17.868596 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:54:17.868567    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f87a54c_e893_4a9e_9f61_b2922fc8f4e5.slice/crio-ac374c4a5fff33e83ee356758ead06ae5beac3f869093c32879c5d7819970bfe WatchSource:0}: Error finding container ac374c4a5fff33e83ee356758ead06ae5beac3f869093c32879c5d7819970bfe: Status 404 returned error can't find the container with id ac374c4a5fff33e83ee356758ead06ae5beac3f869093c32879c5d7819970bfe
Sep 18 15:54:17.870962 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.870932    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:54:17.871050 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.870994    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:54:17.871050 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:17.871022    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:54:18.305923 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:18.305893    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" event={"ID":"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5","Type":"ContainerStarted","Data":"053345fbe629ecbc7d225f14021eb4e693bdf47ee747dea6a06ec1d7152c8b46"}
Sep 18 15:54:18.306303 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:18.305942    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" event={"ID":"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5","Type":"ContainerStarted","Data":"ac374c4a5fff33e83ee356758ead06ae5beac3f869093c32879c5d7819970bfe"}
Sep 18 15:54:18.328423 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:18.328381    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" podStartSLOduration=1.328366308 podStartE2EDuration="1.328366308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:54:18.3274409 +0000 UTC m=+1201.872731576" watchObservedRunningTime="2026-09-18 15:54:18.328366308 +0000 UTC m=+1201.873656981"
Sep 18 15:54:18.388828 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:18.388803    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 15:54:18.393722 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:18.393702    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 15:54:18.592953 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:18.592873    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:18.598086 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:18.598064    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:19.312918 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:19.312886    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:19.313827 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:19.313804    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:54:21.112433 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.112391    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"]
Sep 18 15:54:21.116574 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.116553    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.120510 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.120486    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f14aba\"/\"auth-test-model-f14aba-kserve-self-signed-certs\""
Sep 18 15:54:21.120598 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.120537    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f14aba\"/\"openshift-service-ca.crt\""
Sep 18 15:54:21.120598 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.120584    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-f14aba\"/\"kube-root-ca.crt\""
Sep 18 15:54:21.120723 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.120489    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f14aba\"/\"default-dockercfg-qpkcw\""
Sep 18 15:54:21.125524 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.125501    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"]
Sep 18 15:54:21.157914 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.157890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-model-cache\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.158062 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.157929    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-home\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.158062 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.157970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-tmp-dir\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.158062 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.157985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-dshm\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.158062 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.158031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkh25\" (UniqueName: \"kubernetes.io/projected/efccf070-2d1b-4446-9a3b-9dc0e30f8260-kube-api-access-dkh25\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.158289 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.158075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efccf070-2d1b-4446-9a3b-9dc0e30f8260-tls-certs\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.258815 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.258784    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-model-cache\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.258984 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.258823    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-home\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.258984 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.258858    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-tmp-dir\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.258984 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.258874    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-dshm\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.258984 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.258918    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkh25\" (UniqueName: \"kubernetes.io/projected/efccf070-2d1b-4446-9a3b-9dc0e30f8260-kube-api-access-dkh25\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.258984 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.258933    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efccf070-2d1b-4446-9a3b-9dc0e30f8260-tls-certs\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.259326 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.259225    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-model-cache\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.259326 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.259262    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-home\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.259432 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.259328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-tmp-dir\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.261056 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.261029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-dshm\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.261536 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.261517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efccf070-2d1b-4446-9a3b-9dc0e30f8260-tls-certs\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.267108 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.267084    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkh25\" (UniqueName: \"kubernetes.io/projected/efccf070-2d1b-4446-9a3b-9dc0e30f8260-kube-api-access-dkh25\") pod \"auth-test-model-f14aba-kserve-5d568b5c47-zxnv2\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.427638 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.427605    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:21.712232 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:21.712203    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"]
Sep 18 15:54:21.713894 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:54:21.713849    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefccf070_2d1b_4446_9a3b_9dc0e30f8260.slice/crio-f915fbc8370e58466fde9fcf98b8548beb633c01204b2adfc15118edfa248908 WatchSource:0}: Error finding container f915fbc8370e58466fde9fcf98b8548beb633c01204b2adfc15118edfa248908: Status 404 returned error can't find the container with id f915fbc8370e58466fde9fcf98b8548beb633c01204b2adfc15118edfa248908
Sep 18 15:54:22.334536 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:22.334482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2" event={"ID":"efccf070-2d1b-4446-9a3b-9dc0e30f8260","Type":"ContainerStarted","Data":"fbfd179dcbe40805291533bc994f7f1b3de65829da0d93ff97022590f8ff8cf8"}
Sep 18 15:54:22.334536 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:22.334543    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2" event={"ID":"efccf070-2d1b-4446-9a3b-9dc0e30f8260","Type":"ContainerStarted","Data":"f915fbc8370e58466fde9fcf98b8548beb633c01204b2adfc15118edfa248908"}
Sep 18 15:54:22.354515 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:22.354463    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2" podStartSLOduration=1.354443679 podStartE2EDuration="1.354443679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:54:22.352363912 +0000 UTC m=+1205.897654596" watchObservedRunningTime="2026-09-18 15:54:22.354443679 +0000 UTC m=+1205.899734354"
Sep 18 15:54:31.428669 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:31.428634    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:31.429086 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:31.428694    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:31.445282 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:31.445254    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:32.400117 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:32.400090    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:54:51.010034 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:51.009988    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9131-655e4d"]
Sep 18 15:54:51.014584 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:51.014553    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-655e4d"
Sep 18 15:54:51.017343 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:51.017314    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-w9zf6\""
Sep 18 15:54:51.020026 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:51.020001    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9131-655e4d"]
Sep 18 15:54:51.081416 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:51.081388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v88bh\" (UniqueName: \"kubernetes.io/projected/17e6f257-95ee-442b-aa39-06f8794178f9-kube-api-access-v88bh\") pod \"test-curl-9131-655e4d\" (UID: \"17e6f257-95ee-442b-aa39-06f8794178f9\") " pod="opendatahub/test-curl-9131-655e4d"
Sep 18 15:54:51.182821 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:51.182791    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v88bh\" (UniqueName: \"kubernetes.io/projected/17e6f257-95ee-442b-aa39-06f8794178f9-kube-api-access-v88bh\") pod \"test-curl-9131-655e4d\" (UID: \"17e6f257-95ee-442b-aa39-06f8794178f9\") " pod="opendatahub/test-curl-9131-655e4d"
Sep 18 15:54:51.192955 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:51.192928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v88bh\" (UniqueName: \"kubernetes.io/projected/17e6f257-95ee-442b-aa39-06f8794178f9-kube-api-access-v88bh\") pod \"test-curl-9131-655e4d\" (UID: \"17e6f257-95ee-442b-aa39-06f8794178f9\") " pod="opendatahub/test-curl-9131-655e4d"
Sep 18 15:54:51.327170 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:51.327080    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-655e4d"
Sep 18 15:54:51.601096 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:51.601006    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9131-655e4d"]
Sep 18 15:54:51.613407 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:54:51.613367    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17e6f257_95ee_442b_aa39_06f8794178f9.slice/crio-61081998ba40a861e582f3f04b10da76869414dea4e1fffe0f2d50e8eac6d944 WatchSource:0}: Error finding container 61081998ba40a861e582f3f04b10da76869414dea4e1fffe0f2d50e8eac6d944: Status 404 returned error can't find the container with id 61081998ba40a861e582f3f04b10da76869414dea4e1fffe0f2d50e8eac6d944
Sep 18 15:54:52.523379 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:52.523346    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9131-655e4d" event={"ID":"17e6f257-95ee-442b-aa39-06f8794178f9","Type":"ContainerStarted","Data":"4753ff27f918056426e04efef4f0e4151f8cc8fa2030de88cbb156bbce28b3de"}
Sep 18 15:54:52.523975 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:52.523398    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9131-655e4d" event={"ID":"17e6f257-95ee-442b-aa39-06f8794178f9","Type":"ContainerStarted","Data":"61081998ba40a861e582f3f04b10da76869414dea4e1fffe0f2d50e8eac6d944"}
Sep 18 15:54:52.538211 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:52.538160    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9131-655e4d" podStartSLOduration=2.538145138 podStartE2EDuration="2.538145138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:54:52.536343838 +0000 UTC m=+1236.081634512" watchObservedRunningTime="2026-09-18 15:54:52.538145138 +0000 UTC m=+1236.083435814"
Sep 18 15:54:53.156335 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:53.156300    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9131-655e4d"]
Sep 18 15:54:53.159307 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:53.159282    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9131-655e4d"]
Sep 18 15:54:53.274259 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:53.274169    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9131-76fa15"]
Sep 18 15:54:53.275381 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:53.275348    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17e6f257-95ee-442b-aa39-06f8794178f9" containerName="test-curl-9131-655e4d"
Sep 18 15:54:53.275381 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:53.275375    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17e6f257-95ee-442b-aa39-06f8794178f9" containerName="test-curl-9131-655e4d"
Sep 18 15:54:53.275624 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:53.275603    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17e6f257-95ee-442b-aa39-06f8794178f9" containerName="test-curl-9131-655e4d"
Sep 18 15:54:53.280241 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:53.280214    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-76fa15"
Sep 18 15:54:53.284888 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:53.284859    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9131-76fa15"]
Sep 18 15:54:53.421851 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:53.421819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8cx8\" (UniqueName: \"kubernetes.io/projected/57be6730-0e93-458c-9ac1-a87e66823e4c-kube-api-access-g8cx8\") pod \"test-curl-9131-76fa15\" (UID: \"57be6730-0e93-458c-9ac1-a87e66823e4c\") " pod="opendatahub/test-curl-9131-76fa15"
Sep 18 15:54:53.522866 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:53.522833    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8cx8\" (UniqueName: \"kubernetes.io/projected/57be6730-0e93-458c-9ac1-a87e66823e4c-kube-api-access-g8cx8\") pod \"test-curl-9131-76fa15\" (UID: \"57be6730-0e93-458c-9ac1-a87e66823e4c\") " pod="opendatahub/test-curl-9131-76fa15"
Sep 18 15:54:53.528172 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:54:53.528105    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9131-655e4d" container="test-curl-9131-655e4d"
Sep 18 15:54:53.532589 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:53.532565    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8cx8\" (UniqueName: \"kubernetes.io/projected/57be6730-0e93-458c-9ac1-a87e66823e4c-kube-api-access-g8cx8\") pod \"test-curl-9131-76fa15\" (UID: \"57be6730-0e93-458c-9ac1-a87e66823e4c\") " pod="opendatahub/test-curl-9131-76fa15"
Sep 18 15:54:53.621139 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:53.621107    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-76fa15"
Sep 18 15:54:53.896015 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:53.895975    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9131-76fa15"]
Sep 18 15:54:54.535733 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:54.535703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9131-76fa15" event={"ID":"57be6730-0e93-458c-9ac1-a87e66823e4c","Type":"ContainerStarted","Data":"94e5b307bd2730a7f05f123c7d62417e8bbd96b2bf07c4c52ea156b38fea2f5a"}
Sep 18 15:54:54.536146 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:54.535744    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9131-76fa15" event={"ID":"57be6730-0e93-458c-9ac1-a87e66823e4c","Type":"ContainerStarted","Data":"f390a4d03e9361fd9eacc2df7f0cd61e755376048ccb5a9ec6e7b33e9692fcce"}
Sep 18 15:54:54.536146 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:54.535831    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9131-655e4d" podUID="17e6f257-95ee-442b-aa39-06f8794178f9" containerName="test-curl-9131-655e4d" containerID="cri-o://4753ff27f918056426e04efef4f0e4151f8cc8fa2030de88cbb156bbce28b3de" gracePeriod=2
Sep 18 15:54:54.552445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:54.552404    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9131-76fa15" podStartSLOduration=1.552390065 podStartE2EDuration="1.552390065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:54:54.550983321 +0000 UTC m=+1238.096273994" watchObservedRunningTime="2026-09-18 15:54:54.552390065 +0000 UTC m=+1238.097680756"
Sep 18 15:54:55.158292 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:55.158261    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9131-76fa15"]
Sep 18 15:54:55.162112 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:55.162084    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9131-76fa15"]
Sep 18 15:54:55.269586 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:55.269546    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9131-0efa96"]
Sep 18 15:54:55.270096 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:55.270076    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57be6730-0e93-458c-9ac1-a87e66823e4c" containerName="test-curl-9131-76fa15"
Sep 18 15:54:55.270210 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:55.270099    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="57be6730-0e93-458c-9ac1-a87e66823e4c" containerName="test-curl-9131-76fa15"
Sep 18 15:54:55.270272 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:55.270256    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="57be6730-0e93-458c-9ac1-a87e66823e4c" containerName="test-curl-9131-76fa15"
Sep 18 15:54:55.273743 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:55.273722    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-0efa96"
Sep 18 15:54:55.279209 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:55.279185    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9131-0efa96"]
Sep 18 15:54:55.340383 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:55.340354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfmz\" (UniqueName: \"kubernetes.io/projected/f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3-kube-api-access-mpfmz\") pod \"test-curl-9131-0efa96\" (UID: \"f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3\") " pod="opendatahub/test-curl-9131-0efa96"
Sep 18 15:54:55.441256 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:55.441226    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfmz\" (UniqueName: \"kubernetes.io/projected/f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3-kube-api-access-mpfmz\") pod \"test-curl-9131-0efa96\" (UID: \"f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3\") " pod="opendatahub/test-curl-9131-0efa96"
Sep 18 15:54:55.449629 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:55.449606    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfmz\" (UniqueName: \"kubernetes.io/projected/f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3-kube-api-access-mpfmz\") pod \"test-curl-9131-0efa96\" (UID: \"f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3\") " pod="opendatahub/test-curl-9131-0efa96"
Sep 18 15:54:55.541006 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:54:55.540932    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9131-76fa15" container="test-curl-9131-76fa15"
Sep 18 15:54:55.584377 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:55.584346    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-0efa96"
Sep 18 15:54:55.882892 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:55.882805    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9131-0efa96"]
Sep 18 15:54:55.885689 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:54:55.885649    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf41f4c2a_8e05_4744_ada5_d3f85b6c4fe3.slice/crio-097c8c0d13c6209482f98a3913e59afb35ddb0aa876d9f5ae4888368f424ea9a WatchSource:0}: Error finding container 097c8c0d13c6209482f98a3913e59afb35ddb0aa876d9f5ae4888368f424ea9a: Status 404 returned error can't find the container with id 097c8c0d13c6209482f98a3913e59afb35ddb0aa876d9f5ae4888368f424ea9a
Sep 18 15:54:56.553315 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:56.553283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9131-0efa96" event={"ID":"f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3","Type":"ContainerStarted","Data":"c840c9229552e7d823f05f4e73a27416fcc6c976b17c0d15337e359019bb1ccc"}
Sep 18 15:54:56.553731 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:56.553336    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9131-0efa96" event={"ID":"f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3","Type":"ContainerStarted","Data":"097c8c0d13c6209482f98a3913e59afb35ddb0aa876d9f5ae4888368f424ea9a"}
Sep 18 15:54:56.553731 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:56.553468    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9131-76fa15" podUID="57be6730-0e93-458c-9ac1-a87e66823e4c" containerName="test-curl-9131-76fa15" containerID="cri-o://94e5b307bd2730a7f05f123c7d62417e8bbd96b2bf07c4c52ea156b38fea2f5a" gracePeriod=2
Sep 18 15:54:56.571277 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:56.571218    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9131-0efa96" podStartSLOduration=1.571198829 podStartE2EDuration="1.571198829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:54:56.567616752 +0000 UTC m=+1240.112907426" watchObservedRunningTime="2026-09-18 15:54:56.571198829 +0000 UTC m=+1240.116489505"
Sep 18 15:54:56.721244 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:56.721216    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-655e4d"
Sep 18 15:54:56.857082 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:56.856990    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v88bh\" (UniqueName: \"kubernetes.io/projected/17e6f257-95ee-442b-aa39-06f8794178f9-kube-api-access-v88bh\") pod \"17e6f257-95ee-442b-aa39-06f8794178f9\" (UID: \"17e6f257-95ee-442b-aa39-06f8794178f9\") "
Sep 18 15:54:56.859166 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:56.859144    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e6f257-95ee-442b-aa39-06f8794178f9-kube-api-access-v88bh" (OuterVolumeSpecName: "kube-api-access-v88bh") pod "17e6f257-95ee-442b-aa39-06f8794178f9" (UID: "17e6f257-95ee-442b-aa39-06f8794178f9"). InnerVolumeSpecName "kube-api-access-v88bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:54:56.958058 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:56.958028    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v88bh\" (UniqueName: \"kubernetes.io/projected/17e6f257-95ee-442b-aa39-06f8794178f9-kube-api-access-v88bh\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:54:57.130491 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.130401    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9131-0efa96"]
Sep 18 15:54:57.135284 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.135252    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9131-0efa96"]
Sep 18 15:54:57.244500 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.244462    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9131-b73aa9"]
Sep 18 15:54:57.244967 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.244949    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3" containerName="test-curl-9131-0efa96"
Sep 18 15:54:57.244967 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.244966    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3" containerName="test-curl-9131-0efa96"
Sep 18 15:54:57.245128 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.245052    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3" containerName="test-curl-9131-0efa96"
Sep 18 15:54:57.248306 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.248285    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-b73aa9"
Sep 18 15:54:57.254114 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.254085    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9131-b73aa9"]
Sep 18 15:54:57.361572 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.361534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glzsn\" (UniqueName: \"kubernetes.io/projected/0a12ff7d-e665-4fc0-8da6-ec8d8060ba04-kube-api-access-glzsn\") pod \"test-curl-9131-b73aa9\" (UID: \"0a12ff7d-e665-4fc0-8da6-ec8d8060ba04\") " pod="opendatahub/test-curl-9131-b73aa9"
Sep 18 15:54:57.463034 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.462991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glzsn\" (UniqueName: \"kubernetes.io/projected/0a12ff7d-e665-4fc0-8da6-ec8d8060ba04-kube-api-access-glzsn\") pod \"test-curl-9131-b73aa9\" (UID: \"0a12ff7d-e665-4fc0-8da6-ec8d8060ba04\") " pod="opendatahub/test-curl-9131-b73aa9"
Sep 18 15:54:57.471076 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.471043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glzsn\" (UniqueName: \"kubernetes.io/projected/0a12ff7d-e665-4fc0-8da6-ec8d8060ba04-kube-api-access-glzsn\") pod \"test-curl-9131-b73aa9\" (UID: \"0a12ff7d-e665-4fc0-8da6-ec8d8060ba04\") " pod="opendatahub/test-curl-9131-b73aa9"
Sep 18 15:54:57.560826 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.560795    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-b73aa9"
Sep 18 15:54:57.561240 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.560960    2561 generic.go:356] "Generic (PLEG): container finished" podID="17e6f257-95ee-442b-aa39-06f8794178f9" containerID="4753ff27f918056426e04efef4f0e4151f8cc8fa2030de88cbb156bbce28b3de" exitCode=137
Sep 18 15:54:57.561240 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.561019    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-655e4d"
Sep 18 15:54:57.561240 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.561049    2561 scope.go:117] "RemoveContainer" containerID="4753ff27f918056426e04efef4f0e4151f8cc8fa2030de88cbb156bbce28b3de"
Sep 18 15:54:57.561428 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:54:57.561411    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9131-0efa96" container="test-curl-9131-0efa96"
Sep 18 15:54:57.576458 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.576406    2561 scope.go:117] "RemoveContainer" containerID="4753ff27f918056426e04efef4f0e4151f8cc8fa2030de88cbb156bbce28b3de"
Sep 18 15:54:57.576753 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:54:57.576727    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4753ff27f918056426e04efef4f0e4151f8cc8fa2030de88cbb156bbce28b3de\": container with ID starting with 4753ff27f918056426e04efef4f0e4151f8cc8fa2030de88cbb156bbce28b3de not found: ID does not exist" containerID="4753ff27f918056426e04efef4f0e4151f8cc8fa2030de88cbb156bbce28b3de"
Sep 18 15:54:57.576843 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.576764    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4753ff27f918056426e04efef4f0e4151f8cc8fa2030de88cbb156bbce28b3de"} err="failed to get container status \"4753ff27f918056426e04efef4f0e4151f8cc8fa2030de88cbb156bbce28b3de\": rpc error: code = NotFound desc = could not find container \"4753ff27f918056426e04efef4f0e4151f8cc8fa2030de88cbb156bbce28b3de\": container with ID starting with 4753ff27f918056426e04efef4f0e4151f8cc8fa2030de88cbb156bbce28b3de not found: ID does not exist"
Sep 18 15:54:57.599911 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:54:57.599872    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9131-655e4d\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="17e6f257-95ee-442b-aa39-06f8794178f9" pod="opendatahub/test-curl-9131-655e4d"
Sep 18 15:54:57.848044 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:57.847959    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9131-b73aa9"]
Sep 18 15:54:57.850441 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:54:57.850416    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a12ff7d_e665_4fc0_8da6_ec8d8060ba04.slice/crio-906883c27e20873f2a129b48a02ccc4a72db625afb5a247dd2e4b84d513df8f5 WatchSource:0}: Error finding container 906883c27e20873f2a129b48a02ccc4a72db625afb5a247dd2e4b84d513df8f5: Status 404 returned error can't find the container with id 906883c27e20873f2a129b48a02ccc4a72db625afb5a247dd2e4b84d513df8f5
Sep 18 15:54:58.378748 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:54:58.378711    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9131-655e4d\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="17e6f257-95ee-442b-aa39-06f8794178f9" pod="opendatahub/test-curl-9131-655e4d"
Sep 18 15:54:58.381463 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:58.381434    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17e6f257-95ee-442b-aa39-06f8794178f9" path="/var/lib/kubelet/pods/17e6f257-95ee-442b-aa39-06f8794178f9/volumes"
Sep 18 15:54:58.568787 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:58.568757    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9131-b73aa9" event={"ID":"0a12ff7d-e665-4fc0-8da6-ec8d8060ba04","Type":"ContainerStarted","Data":"67e539fe250f2be7fa7a1a4096e23ba200a528a453cb34397d8148489593e7b6"}
Sep 18 15:54:58.569095 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:58.568802    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9131-b73aa9" event={"ID":"0a12ff7d-e665-4fc0-8da6-ec8d8060ba04","Type":"ContainerStarted","Data":"906883c27e20873f2a129b48a02ccc4a72db625afb5a247dd2e4b84d513df8f5"}
Sep 18 15:54:58.569095 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:58.568971    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9131-0efa96" podUID="f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3" containerName="test-curl-9131-0efa96" containerID="cri-o://c840c9229552e7d823f05f4e73a27416fcc6c976b17c0d15337e359019bb1ccc" gracePeriod=2
Sep 18 15:54:58.585134 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:58.585090    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-9131-b73aa9" podStartSLOduration=1.585073138 podStartE2EDuration="1.585073138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:54:58.583112231 +0000 UTC m=+1242.128402905" watchObservedRunningTime="2026-09-18 15:54:58.585073138 +0000 UTC m=+1242.130363814"
Sep 18 15:54:58.753056 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:58.753030    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-76fa15"
Sep 18 15:54:58.875769 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:58.875740    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8cx8\" (UniqueName: \"kubernetes.io/projected/57be6730-0e93-458c-9ac1-a87e66823e4c-kube-api-access-g8cx8\") pod \"57be6730-0e93-458c-9ac1-a87e66823e4c\" (UID: \"57be6730-0e93-458c-9ac1-a87e66823e4c\") "
Sep 18 15:54:58.877893 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:58.877863    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57be6730-0e93-458c-9ac1-a87e66823e4c-kube-api-access-g8cx8" (OuterVolumeSpecName: "kube-api-access-g8cx8") pod "57be6730-0e93-458c-9ac1-a87e66823e4c" (UID: "57be6730-0e93-458c-9ac1-a87e66823e4c"). InnerVolumeSpecName "kube-api-access-g8cx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:54:58.976645 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:58.976624    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8cx8\" (UniqueName: \"kubernetes.io/projected/57be6730-0e93-458c-9ac1-a87e66823e4c-kube-api-access-g8cx8\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:54:59.165278 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:59.165242    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9131-b73aa9"]
Sep 18 15:54:59.168108 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:59.168079    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9131-b73aa9"]
Sep 18 15:54:59.576423 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:59.576324    2561 generic.go:356] "Generic (PLEG): container finished" podID="57be6730-0e93-458c-9ac1-a87e66823e4c" containerID="94e5b307bd2730a7f05f123c7d62417e8bbd96b2bf07c4c52ea156b38fea2f5a" exitCode=137
Sep 18 15:54:59.576423 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:59.576369    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-76fa15"
Sep 18 15:54:59.576951 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:59.576428    2561 scope.go:117] "RemoveContainer" containerID="94e5b307bd2730a7f05f123c7d62417e8bbd96b2bf07c4c52ea156b38fea2f5a"
Sep 18 15:54:59.578318 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:59.578298    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerID="053345fbe629ecbc7d225f14021eb4e693bdf47ee747dea6a06ec1d7152c8b46" exitCode=0
Sep 18 15:54:59.578428 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:59.578380    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" event={"ID":"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5","Type":"ContainerDied","Data":"053345fbe629ecbc7d225f14021eb4e693bdf47ee747dea6a06ec1d7152c8b46"}
Sep 18 15:54:59.578554 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:54:59.578540    2561 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-9131-b73aa9" container="test-curl-9131-b73aa9"
Sep 18 15:54:59.578935 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:59.578916    2561 scope.go:117] "RemoveContainer" containerID="053345fbe629ecbc7d225f14021eb4e693bdf47ee747dea6a06ec1d7152c8b46"
Sep 18 15:54:59.580204 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:59.580163    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:54:59.580304 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:59.580253    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:54:59.580304 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:59.580300    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:54:59.586933 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:59.586916    2561 scope.go:117] "RemoveContainer" containerID="94e5b307bd2730a7f05f123c7d62417e8bbd96b2bf07c4c52ea156b38fea2f5a"
Sep 18 15:54:59.587183 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:54:59.587168    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94e5b307bd2730a7f05f123c7d62417e8bbd96b2bf07c4c52ea156b38fea2f5a\": container with ID starting with 94e5b307bd2730a7f05f123c7d62417e8bbd96b2bf07c4c52ea156b38fea2f5a not found: ID does not exist" containerID="94e5b307bd2730a7f05f123c7d62417e8bbd96b2bf07c4c52ea156b38fea2f5a"
Sep 18 15:54:59.587229 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:54:59.587188    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94e5b307bd2730a7f05f123c7d62417e8bbd96b2bf07c4c52ea156b38fea2f5a"} err="failed to get container status \"94e5b307bd2730a7f05f123c7d62417e8bbd96b2bf07c4c52ea156b38fea2f5a\": rpc error: code = NotFound desc = could not find container \"94e5b307bd2730a7f05f123c7d62417e8bbd96b2bf07c4c52ea156b38fea2f5a\": container with ID starting with 94e5b307bd2730a7f05f123c7d62417e8bbd96b2bf07c4c52ea156b38fea2f5a not found: ID does not exist"
Sep 18 15:54:59.630641 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:54:59.630604    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9131-76fa15\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="57be6730-0e93-458c-9ac1-a87e66823e4c" pod="opendatahub/test-curl-9131-76fa15"
Sep 18 15:55:00.381292 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:00.381258    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57be6730-0e93-458c-9ac1-a87e66823e4c" path="/var/lib/kubelet/pods/57be6730-0e93-458c-9ac1-a87e66823e4c/volumes"
Sep 18 15:55:00.473710 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:00.473676    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"]
Sep 18 15:55:00.587690 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:00.587641    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" event={"ID":"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5","Type":"ContainerStarted","Data":"dda14807eaeee411c8a68d85e2f2648b67925c6ae9ee2dc64ccfe4a5946c0afc"}
Sep 18 15:55:00.587981 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:00.587755    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-9131-b73aa9" podUID="0a12ff7d-e665-4fc0-8da6-ec8d8060ba04" containerName="test-curl-9131-b73aa9" containerID="cri-o://67e539fe250f2be7fa7a1a4096e23ba200a528a453cb34397d8148489593e7b6" gracePeriod=2
Sep 18 15:55:00.592878 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:00.592853    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:55:00.594039 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:00.594015    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 18 15:55:00.594136 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:00.594071    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 18 15:55:00.745648 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:00.745622    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-0efa96"
Sep 18 15:55:00.912372 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:00.912259    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpfmz\" (UniqueName: \"kubernetes.io/projected/f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3-kube-api-access-mpfmz\") pod \"f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3\" (UID: \"f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3\") "
Sep 18 15:55:00.915735 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:00.915656    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3-kube-api-access-mpfmz" (OuterVolumeSpecName: "kube-api-access-mpfmz") pod "f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3" (UID: "f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3"). InnerVolumeSpecName "kube-api-access-mpfmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:55:01.013556 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:01.013508    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpfmz\" (UniqueName: \"kubernetes.io/projected/f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3-kube-api-access-mpfmz\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:55:01.593229 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:01.593194    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 18 15:55:01.593812 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:01.593783    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 18 15:55:01.598603 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:01.598563    2561 generic.go:356] "Generic (PLEG): container finished" podID="f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3" containerID="c840c9229552e7d823f05f4e73a27416fcc6c976b17c0d15337e359019bb1ccc" exitCode=137
Sep 18 15:55:01.599506 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:01.599480    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-0efa96"
Sep 18 15:55:01.605105 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:01.605082    2561 scope.go:117] "RemoveContainer" containerID="c840c9229552e7d823f05f4e73a27416fcc6c976b17c0d15337e359019bb1ccc"
Sep 18 15:55:01.620425 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:01.620404    2561 scope.go:117] "RemoveContainer" containerID="c840c9229552e7d823f05f4e73a27416fcc6c976b17c0d15337e359019bb1ccc"
Sep 18 15:55:01.620802 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:55:01.620779    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c840c9229552e7d823f05f4e73a27416fcc6c976b17c0d15337e359019bb1ccc\": container with ID starting with c840c9229552e7d823f05f4e73a27416fcc6c976b17c0d15337e359019bb1ccc not found: ID does not exist" containerID="c840c9229552e7d823f05f4e73a27416fcc6c976b17c0d15337e359019bb1ccc"
Sep 18 15:55:01.620932 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:01.620808    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c840c9229552e7d823f05f4e73a27416fcc6c976b17c0d15337e359019bb1ccc"} err="failed to get container status \"c840c9229552e7d823f05f4e73a27416fcc6c976b17c0d15337e359019bb1ccc\": rpc error: code = NotFound desc = could not find container \"c840c9229552e7d823f05f4e73a27416fcc6c976b17c0d15337e359019bb1ccc\": container with ID starting with c840c9229552e7d823f05f4e73a27416fcc6c976b17c0d15337e359019bb1ccc not found: ID does not exist"
Sep 18 15:55:01.620992 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:55:01.620970    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9131-0efa96\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3" pod="opendatahub/test-curl-9131-0efa96"
Sep 18 15:55:02.381221 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:02.381189    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3" path="/var/lib/kubelet/pods/f41f4c2a-8e05-4744-ada5-d3f85b6c4fe3/volumes"
Sep 18 15:55:02.593655 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:02.593612    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 18 15:55:02.594093 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:02.593715    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 18 15:55:02.819945 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:02.819916    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-b73aa9"
Sep 18 15:55:02.830144 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:02.830121    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glzsn\" (UniqueName: \"kubernetes.io/projected/0a12ff7d-e665-4fc0-8da6-ec8d8060ba04-kube-api-access-glzsn\") pod \"0a12ff7d-e665-4fc0-8da6-ec8d8060ba04\" (UID: \"0a12ff7d-e665-4fc0-8da6-ec8d8060ba04\") "
Sep 18 15:55:02.832137 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:02.832108    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a12ff7d-e665-4fc0-8da6-ec8d8060ba04-kube-api-access-glzsn" (OuterVolumeSpecName: "kube-api-access-glzsn") pod "0a12ff7d-e665-4fc0-8da6-ec8d8060ba04" (UID: "0a12ff7d-e665-4fc0-8da6-ec8d8060ba04"). InnerVolumeSpecName "kube-api-access-glzsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:55:02.931521 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:02.931491    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glzsn\" (UniqueName: \"kubernetes.io/projected/0a12ff7d-e665-4fc0-8da6-ec8d8060ba04-kube-api-access-glzsn\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:55:03.593533 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:03.593488    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 18 15:55:03.593768 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:03.593558    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 18 15:55:03.617482 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:03.617452    2561 generic.go:356] "Generic (PLEG): container finished" podID="0a12ff7d-e665-4fc0-8da6-ec8d8060ba04" containerID="67e539fe250f2be7fa7a1a4096e23ba200a528a453cb34397d8148489593e7b6" exitCode=137
Sep 18 15:55:03.617635 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:03.617495    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9131-b73aa9"
Sep 18 15:55:03.617635 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:03.617563    2561 scope.go:117] "RemoveContainer" containerID="67e539fe250f2be7fa7a1a4096e23ba200a528a453cb34397d8148489593e7b6"
Sep 18 15:55:03.627468 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:03.627452    2561 scope.go:117] "RemoveContainer" containerID="67e539fe250f2be7fa7a1a4096e23ba200a528a453cb34397d8148489593e7b6"
Sep 18 15:55:03.627737 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:55:03.627718    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67e539fe250f2be7fa7a1a4096e23ba200a528a453cb34397d8148489593e7b6\": container with ID starting with 67e539fe250f2be7fa7a1a4096e23ba200a528a453cb34397d8148489593e7b6 not found: ID does not exist" containerID="67e539fe250f2be7fa7a1a4096e23ba200a528a453cb34397d8148489593e7b6"
Sep 18 15:55:03.627813 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:03.627745    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67e539fe250f2be7fa7a1a4096e23ba200a528a453cb34397d8148489593e7b6"} err="failed to get container status \"67e539fe250f2be7fa7a1a4096e23ba200a528a453cb34397d8148489593e7b6\": rpc error: code = NotFound desc = could not find container \"67e539fe250f2be7fa7a1a4096e23ba200a528a453cb34397d8148489593e7b6\": container with ID starting with 67e539fe250f2be7fa7a1a4096e23ba200a528a453cb34397d8148489593e7b6 not found: ID does not exist"
Sep 18 15:55:03.628138 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:55:03.628118    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-9131-b73aa9\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="0a12ff7d-e665-4fc0-8da6-ec8d8060ba04" pod="opendatahub/test-curl-9131-b73aa9"
Sep 18 15:55:04.313081 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:04.312985    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:55:04.381142 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:04.381112    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a12ff7d-e665-4fc0-8da6-ec8d8060ba04" path="/var/lib/kubelet/pods/0a12ff7d-e665-4fc0-8da6-ec8d8060ba04/volumes"
Sep 18 15:55:04.593089 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:04.592999    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 18 15:55:04.593342 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:04.593311    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 18 15:55:05.593715 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:05.593676    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Sep 18 15:55:05.594313 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:05.594278    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Sep 18 15:55:06.596781 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:06.596744    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:55:06.597813 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:06.597786    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 15:55:14.182476 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.182301    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"]
Sep 18 15:55:14.182981 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.182566    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc" podUID="15c7ffbc-8e83-494b-a6c8-eefc7761b2eb" containerName="payload-processing" containerID="cri-o://717aaff634b7e511d37c6a7b10da109b73c0c036571d98f2f081b9ff2f9e4231" gracePeriod=30
Sep 18 15:55:14.192735 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.192705    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"]
Sep 18 15:55:14.192962 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.192940    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2" podUID="cb58ac36-8738-41cc-9cfc-c24652ddacd2" containerName="payload-pre-processing" containerID="cri-o://9b55beaac3377ef2a37f3dd567057a53ef9237661becff68cd837fe214ed57b3" gracePeriod=30
Sep 18 15:55:14.473945 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.473925    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"
Sep 18 15:55:14.477494 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.477476    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"
Sep 18 15:55:14.539061 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.539031    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c7ffbc-8e83-494b-a6c8-eefc7761b2eb-plugins-config-volume\") pod \"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb\" (UID: \"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb\") "
Sep 18 15:55:14.539252 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.539234    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxczk\" (UniqueName: \"kubernetes.io/projected/cb58ac36-8738-41cc-9cfc-c24652ddacd2-kube-api-access-cxczk\") pod \"cb58ac36-8738-41cc-9cfc-c24652ddacd2\" (UID: \"cb58ac36-8738-41cc-9cfc-c24652ddacd2\") "
Sep 18 15:55:14.539320 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.539280    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp7lp\" (UniqueName: \"kubernetes.io/projected/15c7ffbc-8e83-494b-a6c8-eefc7761b2eb-kube-api-access-zp7lp\") pod \"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb\" (UID: \"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb\") "
Sep 18 15:55:14.539385 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.539353    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb58ac36-8738-41cc-9cfc-c24652ddacd2-plugins-config-volume\") pod \"cb58ac36-8738-41cc-9cfc-c24652ddacd2\" (UID: \"cb58ac36-8738-41cc-9cfc-c24652ddacd2\") "
Sep 18 15:55:14.539470 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.539445    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c7ffbc-8e83-494b-a6c8-eefc7761b2eb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "15c7ffbc-8e83-494b-a6c8-eefc7761b2eb" (UID: "15c7ffbc-8e83-494b-a6c8-eefc7761b2eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:55:14.539758 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.539731    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c7ffbc-8e83-494b-a6c8-eefc7761b2eb-plugins-config-volume\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:55:14.540133 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.540107    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb58ac36-8738-41cc-9cfc-c24652ddacd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb58ac36-8738-41cc-9cfc-c24652ddacd2" (UID: "cb58ac36-8738-41cc-9cfc-c24652ddacd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:55:14.544533 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.544504    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c7ffbc-8e83-494b-a6c8-eefc7761b2eb-kube-api-access-zp7lp" (OuterVolumeSpecName: "kube-api-access-zp7lp") pod "15c7ffbc-8e83-494b-a6c8-eefc7761b2eb" (UID: "15c7ffbc-8e83-494b-a6c8-eefc7761b2eb"). InnerVolumeSpecName "kube-api-access-zp7lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:55:14.545707 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.545631    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb58ac36-8738-41cc-9cfc-c24652ddacd2-kube-api-access-cxczk" (OuterVolumeSpecName: "kube-api-access-cxczk") pod "cb58ac36-8738-41cc-9cfc-c24652ddacd2" (UID: "cb58ac36-8738-41cc-9cfc-c24652ddacd2"). InnerVolumeSpecName "kube-api-access-cxczk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:55:14.640601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.640571    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxczk\" (UniqueName: \"kubernetes.io/projected/cb58ac36-8738-41cc-9cfc-c24652ddacd2-kube-api-access-cxczk\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:55:14.640601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.640594    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp7lp\" (UniqueName: \"kubernetes.io/projected/15c7ffbc-8e83-494b-a6c8-eefc7761b2eb-kube-api-access-zp7lp\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:55:14.640601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.640605    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb58ac36-8738-41cc-9cfc-c24652ddacd2-plugins-config-volume\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:55:14.678484 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.678457    2561 generic.go:356] "Generic (PLEG): container finished" podID="cb58ac36-8738-41cc-9cfc-c24652ddacd2" containerID="9b55beaac3377ef2a37f3dd567057a53ef9237661becff68cd837fe214ed57b3" exitCode=0
Sep 18 15:55:14.678610 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.678502    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"
Sep 18 15:55:14.678610 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.678551    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2" event={"ID":"cb58ac36-8738-41cc-9cfc-c24652ddacd2","Type":"ContainerDied","Data":"9b55beaac3377ef2a37f3dd567057a53ef9237661becff68cd837fe214ed57b3"}
Sep 18 15:55:14.678610 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.678604    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2" event={"ID":"cb58ac36-8738-41cc-9cfc-c24652ddacd2","Type":"ContainerDied","Data":"d72194ceb2b4558f9b509a14a8dab2094896018f8da93543377eb268c9e7bd2f"}
Sep 18 15:55:14.678790 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.678630    2561 scope.go:117] "RemoveContainer" containerID="9b55beaac3377ef2a37f3dd567057a53ef9237661becff68cd837fe214ed57b3"
Sep 18 15:55:14.680900 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.680874    2561 generic.go:356] "Generic (PLEG): container finished" podID="15c7ffbc-8e83-494b-a6c8-eefc7761b2eb" containerID="717aaff634b7e511d37c6a7b10da109b73c0c036571d98f2f081b9ff2f9e4231" exitCode=0
Sep 18 15:55:14.681019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.680913    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"
Sep 18 15:55:14.681019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.680942    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc" event={"ID":"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb","Type":"ContainerDied","Data":"717aaff634b7e511d37c6a7b10da109b73c0c036571d98f2f081b9ff2f9e4231"}
Sep 18 15:55:14.681019 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.680987    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc" event={"ID":"15c7ffbc-8e83-494b-a6c8-eefc7761b2eb","Type":"ContainerDied","Data":"834cf388cfc5e7c875c687c2d3ee295bd99eebe8f07fe6b2d319c85b6274efa6"}
Sep 18 15:55:14.688567 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.688548    2561 scope.go:117] "RemoveContainer" containerID="9b55beaac3377ef2a37f3dd567057a53ef9237661becff68cd837fe214ed57b3"
Sep 18 15:55:14.688910 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:55:14.688890    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b55beaac3377ef2a37f3dd567057a53ef9237661becff68cd837fe214ed57b3\": container with ID starting with 9b55beaac3377ef2a37f3dd567057a53ef9237661becff68cd837fe214ed57b3 not found: ID does not exist" containerID="9b55beaac3377ef2a37f3dd567057a53ef9237661becff68cd837fe214ed57b3"
Sep 18 15:55:14.689016 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.688916    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b55beaac3377ef2a37f3dd567057a53ef9237661becff68cd837fe214ed57b3"} err="failed to get container status \"9b55beaac3377ef2a37f3dd567057a53ef9237661becff68cd837fe214ed57b3\": rpc error: code = NotFound desc = could not find container \"9b55beaac3377ef2a37f3dd567057a53ef9237661becff68cd837fe214ed57b3\": container with ID starting with 9b55beaac3377ef2a37f3dd567057a53ef9237661becff68cd837fe214ed57b3 not found: ID does not exist"
Sep 18 15:55:14.689016 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.688932    2561 scope.go:117] "RemoveContainer" containerID="717aaff634b7e511d37c6a7b10da109b73c0c036571d98f2f081b9ff2f9e4231"
Sep 18 15:55:14.697726 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.697704    2561 scope.go:117] "RemoveContainer" containerID="717aaff634b7e511d37c6a7b10da109b73c0c036571d98f2f081b9ff2f9e4231"
Sep 18 15:55:14.697987 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:55:14.697972    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717aaff634b7e511d37c6a7b10da109b73c0c036571d98f2f081b9ff2f9e4231\": container with ID starting with 717aaff634b7e511d37c6a7b10da109b73c0c036571d98f2f081b9ff2f9e4231 not found: ID does not exist" containerID="717aaff634b7e511d37c6a7b10da109b73c0c036571d98f2f081b9ff2f9e4231"
Sep 18 15:55:14.698041 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.697993    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717aaff634b7e511d37c6a7b10da109b73c0c036571d98f2f081b9ff2f9e4231"} err="failed to get container status \"717aaff634b7e511d37c6a7b10da109b73c0c036571d98f2f081b9ff2f9e4231\": rpc error: code = NotFound desc = could not find container \"717aaff634b7e511d37c6a7b10da109b73c0c036571d98f2f081b9ff2f9e4231\": container with ID starting with 717aaff634b7e511d37c6a7b10da109b73c0c036571d98f2f081b9ff2f9e4231 not found: ID does not exist"
Sep 18 15:55:14.703500 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.703480    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"]
Sep 18 15:55:14.709242 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.709221    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-9792ff-5d886cb4c8-62nb2"]
Sep 18 15:55:14.720409 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.720388    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"]
Sep 18 15:55:14.729357 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:14.729340    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-9792ff-74f66fbbbc-9tqqc"]
Sep 18 15:55:15.493681 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.493606    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7" podUID="0928eec3-5a76-414f-8858-2505f8bedbee" containerName="main" containerID="cri-o://f0b1374635a9ce2474b398ca55c6abff81affe19a2b38c52338f3ed806e3d0c6" gracePeriod=45
Sep 18 15:55:15.690017 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.689977    2561 generic.go:356] "Generic (PLEG): container finished" podID="0928eec3-5a76-414f-8858-2505f8bedbee" containerID="f0b1374635a9ce2474b398ca55c6abff81affe19a2b38c52338f3ed806e3d0c6" exitCode=0
Sep 18 15:55:15.690153 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.690046    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7" event={"ID":"0928eec3-5a76-414f-8858-2505f8bedbee","Type":"ContainerDied","Data":"f0b1374635a9ce2474b398ca55c6abff81affe19a2b38c52338f3ed806e3d0c6"}
Sep 18 15:55:15.751341 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.751282    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:55:15.852890 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.852856    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0928eec3-5a76-414f-8858-2505f8bedbee-tls-certs\") pod \"0928eec3-5a76-414f-8858-2505f8bedbee\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") "
Sep 18 15:55:15.853065 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.852951    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-dshm\") pod \"0928eec3-5a76-414f-8858-2505f8bedbee\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") "
Sep 18 15:55:15.853065 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.853003    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-tmp-dir\") pod \"0928eec3-5a76-414f-8858-2505f8bedbee\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") "
Sep 18 15:55:15.853065 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.853063    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-home\") pod \"0928eec3-5a76-414f-8858-2505f8bedbee\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") "
Sep 18 15:55:15.853223 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.853158    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk875\" (UniqueName: \"kubernetes.io/projected/0928eec3-5a76-414f-8858-2505f8bedbee-kube-api-access-tk875\") pod \"0928eec3-5a76-414f-8858-2505f8bedbee\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") "
Sep 18 15:55:15.853282 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.853221    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-model-cache\") pod \"0928eec3-5a76-414f-8858-2505f8bedbee\" (UID: \"0928eec3-5a76-414f-8858-2505f8bedbee\") "
Sep 18 15:55:15.853381 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.853328    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0928eec3-5a76-414f-8858-2505f8bedbee" (UID: "0928eec3-5a76-414f-8858-2505f8bedbee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:15.853445 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.853407    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-home" (OuterVolumeSpecName: "home") pod "0928eec3-5a76-414f-8858-2505f8bedbee" (UID: "0928eec3-5a76-414f-8858-2505f8bedbee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:15.853575 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.853549    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-model-cache" (OuterVolumeSpecName: "model-cache") pod "0928eec3-5a76-414f-8858-2505f8bedbee" (UID: "0928eec3-5a76-414f-8858-2505f8bedbee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:15.853759 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.853727    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.853759 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.853752    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.853929 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.853792    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.855343 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.855302    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-dshm" (OuterVolumeSpecName: "dshm") pod "0928eec3-5a76-414f-8858-2505f8bedbee" (UID: "0928eec3-5a76-414f-8858-2505f8bedbee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:55:15.855444 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.855408    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0928eec3-5a76-414f-8858-2505f8bedbee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0928eec3-5a76-414f-8858-2505f8bedbee" (UID: "0928eec3-5a76-414f-8858-2505f8bedbee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:55:15.855896 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.855874    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0928eec3-5a76-414f-8858-2505f8bedbee-kube-api-access-tk875" (OuterVolumeSpecName: "kube-api-access-tk875") pod "0928eec3-5a76-414f-8858-2505f8bedbee" (UID: "0928eec3-5a76-414f-8858-2505f8bedbee"). InnerVolumeSpecName "kube-api-access-tk875". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:55:15.955163 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.955134    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0928eec3-5a76-414f-8858-2505f8bedbee-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.955163 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.955161    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0928eec3-5a76-414f-8858-2505f8bedbee-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:55:15.955335 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:15.955173    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk875\" (UniqueName: \"kubernetes.io/projected/0928eec3-5a76-414f-8858-2505f8bedbee-kube-api-access-tk875\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:55:16.379923 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:16.379889    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15c7ffbc-8e83-494b-a6c8-eefc7761b2eb" path="/var/lib/kubelet/pods/15c7ffbc-8e83-494b-a6c8-eefc7761b2eb/volumes"
Sep 18 15:55:16.380349 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:16.380328    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb58ac36-8738-41cc-9cfc-c24652ddacd2" path="/var/lib/kubelet/pods/cb58ac36-8738-41cc-9cfc-c24652ddacd2/volumes"
Sep 18 15:55:16.699330 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:16.699301    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"
Sep 18 15:55:16.699782 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:16.699350    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7" event={"ID":"0928eec3-5a76-414f-8858-2505f8bedbee","Type":"ContainerDied","Data":"8e63a5dc5a32b6e69fbe1791a946246bd33b95edeffe87849bacc85b983fdff7"}
Sep 18 15:55:16.699782 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:16.699406    2561 scope.go:117] "RemoveContainer" containerID="f0b1374635a9ce2474b398ca55c6abff81affe19a2b38c52338f3ed806e3d0c6"
Sep 18 15:55:16.716012 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:16.715990    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"]
Sep 18 15:55:16.720388 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:16.720358    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-9792ff/premium-w3-9792ff-kserve-6c775758cb-jrjl7"]
Sep 18 15:55:18.381405 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:18.381378    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0928eec3-5a76-414f-8858-2505f8bedbee" path="/var/lib/kubelet/pods/0928eec3-5a76-414f-8858-2505f8bedbee/volumes"
Sep 18 15:55:42.126249 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.126215    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"]
Sep 18 15:55:42.126721 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.126682    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb58ac36-8738-41cc-9cfc-c24652ddacd2" containerName="payload-pre-processing"
Sep 18 15:55:42.126721 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.126695    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb58ac36-8738-41cc-9cfc-c24652ddacd2" containerName="payload-pre-processing"
Sep 18 15:55:42.126721 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.126707    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0928eec3-5a76-414f-8858-2505f8bedbee" containerName="main"
Sep 18 15:55:42.126721 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.126713    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0928eec3-5a76-414f-8858-2505f8bedbee" containerName="main"
Sep 18 15:55:42.126948 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.126725    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a12ff7d-e665-4fc0-8da6-ec8d8060ba04" containerName="test-curl-9131-b73aa9"
Sep 18 15:55:42.126948 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.126734    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a12ff7d-e665-4fc0-8da6-ec8d8060ba04" containerName="test-curl-9131-b73aa9"
Sep 18 15:55:42.126948 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.126754    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15c7ffbc-8e83-494b-a6c8-eefc7761b2eb" containerName="payload-processing"
Sep 18 15:55:42.126948 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.126760    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="15c7ffbc-8e83-494b-a6c8-eefc7761b2eb" containerName="payload-processing"
Sep 18 15:55:42.126948 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.126839    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0928eec3-5a76-414f-8858-2505f8bedbee" containerName="main"
Sep 18 15:55:42.126948 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.126846    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a12ff7d-e665-4fc0-8da6-ec8d8060ba04" containerName="test-curl-9131-b73aa9"
Sep 18 15:55:42.126948 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.126853    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb58ac36-8738-41cc-9cfc-c24652ddacd2" containerName="payload-pre-processing"
Sep 18 15:55:42.126948 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.126862    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="15c7ffbc-8e83-494b-a6c8-eefc7761b2eb" containerName="payload-processing"
Sep 18 15:55:42.130133 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.130114    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.133189 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.133161    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4e81aa\"/\"openshift-service-ca.crt\""
Sep 18 15:55:42.133296 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.133203    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4e81aa\"/\"kube-root-ca.crt\""
Sep 18 15:55:42.134228 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.134203    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4e81aa\"/\"default-dockercfg-cm5c9\""
Sep 18 15:55:42.134324 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.134275    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4e81aa\"/\"auth-test-model-4e81aa-kserve-self-signed-certs\""
Sep 18 15:55:42.140155 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.140131    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"]
Sep 18 15:55:42.311432 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.311403    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrt8n\" (UniqueName: \"kubernetes.io/projected/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-kube-api-access-vrt8n\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.311583 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.311470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-home\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.311583 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.311528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-tls-certs\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.311583 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.311548    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-dshm\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.311583 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.311579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-tmp-dir\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.311766 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.311595    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-model-cache\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.412470 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.412401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrt8n\" (UniqueName: \"kubernetes.io/projected/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-kube-api-access-vrt8n\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.412470 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.412448    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-home\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.412688 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.412497    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-tls-certs\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.412688 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.412511    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-dshm\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.412688 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.412529    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-tmp-dir\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.412688 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.412618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-model-cache\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.413027 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.413005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-home\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.413109 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.413026    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-model-cache\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.413109 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.413069    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-tmp-dir\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.414753 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.414734    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-dshm\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.415020 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.415001    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-tls-certs\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.421012 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.420992    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrt8n\" (UniqueName: \"kubernetes.io/projected/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-kube-api-access-vrt8n\") pod \"auth-test-model-4e81aa-kserve-76df8dd787-hxz6c\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.440435 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.440417    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:42.722793 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.722770    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"]
Sep 18 15:55:42.725340 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:55:42.725306    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41fea5a4_039b_4977_9d62_e4fef8ffc2d1.slice/crio-febde80a1b78a36b83451d3e2381cd06094ba0ece92ddbc5d28ea1fa4748729e WatchSource:0}: Error finding container febde80a1b78a36b83451d3e2381cd06094ba0ece92ddbc5d28ea1fa4748729e: Status 404 returned error can't find the container with id febde80a1b78a36b83451d3e2381cd06094ba0ece92ddbc5d28ea1fa4748729e
Sep 18 15:55:42.844486 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.844450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c" event={"ID":"41fea5a4-039b-4977-9d62-e4fef8ffc2d1","Type":"ContainerStarted","Data":"7dca3f110f0623a448a75bfb9829e276a8f0260004c28a7e935c1d41a427f808"}
Sep 18 15:55:42.844619 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.844500    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c" event={"ID":"41fea5a4-039b-4977-9d62-e4fef8ffc2d1","Type":"ContainerStarted","Data":"febde80a1b78a36b83451d3e2381cd06094ba0ece92ddbc5d28ea1fa4748729e"}
Sep 18 15:55:42.881963 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:42.881914    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c" podStartSLOduration=0.881894372 podStartE2EDuration="881.894372ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:55:42.880157394 +0000 UTC m=+1286.425448080" watchObservedRunningTime="2026-09-18 15:55:42.881894372 +0000 UTC m=+1286.427185046"
Sep 18 15:55:52.440820 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:52.440788    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:52.441187 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:52.441026    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:52.453350 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:52.453323    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:55:52.909202 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:55:52.909124    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:56:06.028180 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.028147    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"]
Sep 18 15:56:06.031900 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.031883    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.034848 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.034820    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c56bfef6\"/\"e2e-lifecycle-model-c56bfef6-kserve-self-signed-certs\""
Sep 18 15:56:06.035912 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.035889    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c56bfef6\"/\"openshift-service-ca.crt\""
Sep 18 15:56:06.036038 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.035888    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-c56bfef6\"/\"kube-root-ca.crt\""
Sep 18 15:56:06.036038 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.035965    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-c56bfef6\"/\"default-dockercfg-n78k4\""
Sep 18 15:56:06.044935 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.044905    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"]
Sep 18 15:56:06.142112 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.142081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-home\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.142280 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.142143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-dshm\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.142280 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.142203    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zksr\" (UniqueName: \"kubernetes.io/projected/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-kube-api-access-6zksr\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.142280 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.142249    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-model-cache\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.142280 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.142268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-tls-certs\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.142418 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.142331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-tmp-dir\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.243308 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.243276    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-dshm\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.243483 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.243321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zksr\" (UniqueName: \"kubernetes.io/projected/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-kube-api-access-6zksr\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.243483 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.243438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-model-cache\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.243483 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.243473    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-tls-certs\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.243710 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.243518    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-tmp-dir\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.243710 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.243638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-home\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.243894 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.243874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-model-cache\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.243959 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.243938    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-tmp-dir\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.243997 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.243976    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-home\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.245644 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.245622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-dshm\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.245976 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.245960    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-tls-certs\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.251943 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.251926    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zksr\" (UniqueName: \"kubernetes.io/projected/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-kube-api-access-6zksr\") pod \"e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") " pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.341989 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.341913    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:06.625917 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.625890    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"]
Sep 18 15:56:06.626467 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:56:06.626436    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d7f49e1_a73c_4b4f_a86f_6adc7e31f104.slice/crio-6c614f111cd80d6840dfa2d019e199dd17fc17b85eef7b4455778ded84ceb559 WatchSource:0}: Error finding container 6c614f111cd80d6840dfa2d019e199dd17fc17b85eef7b4455778ded84ceb559: Status 404 returned error can't find the container with id 6c614f111cd80d6840dfa2d019e199dd17fc17b85eef7b4455778ded84ceb559
Sep 18 15:56:06.973063 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.973028    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x" event={"ID":"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104","Type":"ContainerStarted","Data":"6ec4f567de81530a0ad912b0b6f99ef998fe459aebc8359b72f02407d8867ac8"}
Sep 18 15:56:06.973279 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.973071    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x" event={"ID":"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104","Type":"ContainerStarted","Data":"6c614f111cd80d6840dfa2d019e199dd17fc17b85eef7b4455778ded84ceb559"}
Sep 18 15:56:06.997615 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:06.997566    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x" podStartSLOduration=0.997549531 podStartE2EDuration="997.549531ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:56:06.99605398 +0000 UTC m=+1310.541344655" watchObservedRunningTime="2026-09-18 15:56:06.997549531 +0000 UTC m=+1310.542840221"
Sep 18 15:56:16.342219 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:16.342176    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:16.342841 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:16.342254    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:16.354848 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:16.354821    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:56:17.036337 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:56:17.036305    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:57:23.127385 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:23.127345    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"]
Sep 18 15:57:38.146564 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.146500    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x" podUID="4d7f49e1-a73c-4b4f-a86f-6adc7e31f104" containerName="main" containerID="cri-o://6ec4f567de81530a0ad912b0b6f99ef998fe459aebc8359b72f02407d8867ac8" gracePeriod=45
Sep 18 15:57:38.400208 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.400151    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:57:38.470210 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.470176    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-tmp-dir\") pod \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") "
Sep 18 15:57:38.470389 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.470252    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zksr\" (UniqueName: \"kubernetes.io/projected/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-kube-api-access-6zksr\") pod \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") "
Sep 18 15:57:38.470389 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.470290    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-model-cache\") pod \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") "
Sep 18 15:57:38.470389 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.470316    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-dshm\") pod \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") "
Sep 18 15:57:38.470389 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.470351    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-home\") pod \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") "
Sep 18 15:57:38.470625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.470429    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-tls-certs\") pod \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\" (UID: \"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104\") "
Sep 18 15:57:38.470625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.470465    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d7f49e1-a73c-4b4f-a86f-6adc7e31f104" (UID: "4d7f49e1-a73c-4b4f-a86f-6adc7e31f104"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:38.470625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.470566    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d7f49e1-a73c-4b4f-a86f-6adc7e31f104" (UID: "4d7f49e1-a73c-4b4f-a86f-6adc7e31f104"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:38.470625 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.470604    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-home" (OuterVolumeSpecName: "home") pod "4d7f49e1-a73c-4b4f-a86f-6adc7e31f104" (UID: "4d7f49e1-a73c-4b4f-a86f-6adc7e31f104"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:38.470890 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.470866    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:38.470956 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.470892    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:38.470956 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.470905    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:38.472170 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.472138    2561 generic.go:356] "Generic (PLEG): container finished" podID="4d7f49e1-a73c-4b4f-a86f-6adc7e31f104" containerID="6ec4f567de81530a0ad912b0b6f99ef998fe459aebc8359b72f02407d8867ac8" exitCode=0
Sep 18 15:57:38.472309 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.472201    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"
Sep 18 15:57:38.472309 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.472222    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x" event={"ID":"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104","Type":"ContainerDied","Data":"6ec4f567de81530a0ad912b0b6f99ef998fe459aebc8359b72f02407d8867ac8"}
Sep 18 15:57:38.472309 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.472259    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x" event={"ID":"4d7f49e1-a73c-4b4f-a86f-6adc7e31f104","Type":"ContainerDied","Data":"6c614f111cd80d6840dfa2d019e199dd17fc17b85eef7b4455778ded84ceb559"}
Sep 18 15:57:38.472309 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.472283    2561 scope.go:117] "RemoveContainer" containerID="6ec4f567de81530a0ad912b0b6f99ef998fe459aebc8359b72f02407d8867ac8"
Sep 18 15:57:38.472888 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.472864    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-dshm" (OuterVolumeSpecName: "dshm") pod "4d7f49e1-a73c-4b4f-a86f-6adc7e31f104" (UID: "4d7f49e1-a73c-4b4f-a86f-6adc7e31f104"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:38.473058 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.473030    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d7f49e1-a73c-4b4f-a86f-6adc7e31f104" (UID: "4d7f49e1-a73c-4b4f-a86f-6adc7e31f104"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:57:38.473181 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.473063    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-kube-api-access-6zksr" (OuterVolumeSpecName: "kube-api-access-6zksr") pod "4d7f49e1-a73c-4b4f-a86f-6adc7e31f104" (UID: "4d7f49e1-a73c-4b4f-a86f-6adc7e31f104"). InnerVolumeSpecName "kube-api-access-6zksr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:57:38.494309 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.494291    2561 scope.go:117] "RemoveContainer" containerID="6ec4f567de81530a0ad912b0b6f99ef998fe459aebc8359b72f02407d8867ac8"
Sep 18 15:57:38.494633 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:57:38.494609    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec4f567de81530a0ad912b0b6f99ef998fe459aebc8359b72f02407d8867ac8\": container with ID starting with 6ec4f567de81530a0ad912b0b6f99ef998fe459aebc8359b72f02407d8867ac8 not found: ID does not exist" containerID="6ec4f567de81530a0ad912b0b6f99ef998fe459aebc8359b72f02407d8867ac8"
Sep 18 15:57:38.494720 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.494642    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec4f567de81530a0ad912b0b6f99ef998fe459aebc8359b72f02407d8867ac8"} err="failed to get container status \"6ec4f567de81530a0ad912b0b6f99ef998fe459aebc8359b72f02407d8867ac8\": rpc error: code = NotFound desc = could not find container \"6ec4f567de81530a0ad912b0b6f99ef998fe459aebc8359b72f02407d8867ac8\": container with ID starting with 6ec4f567de81530a0ad912b0b6f99ef998fe459aebc8359b72f02407d8867ac8 not found: ID does not exist"
Sep 18 15:57:38.571639 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.571602    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zksr\" (UniqueName: \"kubernetes.io/projected/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-kube-api-access-6zksr\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:38.571639 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.571638    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:38.571823 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.571649    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:38.793783 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.793759    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"]
Sep 18 15:57:38.798265 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:38.798240    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-c56bfef6/e2e-lifecycle-model-c56bfef6-kserve-55d9dd5dbc-7td8x"]
Sep 18 15:57:39.116545 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:39.116462    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"]
Sep 18 15:57:39.163690 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:39.163650    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"]
Sep 18 15:57:40.380247 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:40.380220    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d7f49e1-a73c-4b4f-a86f-6adc7e31f104" path="/var/lib/kubelet/pods/4d7f49e1-a73c-4b4f-a86f-6adc7e31f104/volumes"
Sep 18 15:57:50.064467 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.064431    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"]
Sep 18 15:57:50.064978 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.064746    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8" podUID="0bc8396a-f6c2-4385-ac55-19e2c982772a" containerName="payload-processing" containerID="cri-o://aec9978a80b919d6daa5aa4182c2e055c4147826b24e2aa158faef913c283f8e" gracePeriod=30
Sep 18 15:57:50.320930 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.320869    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"
Sep 18 15:57:50.382742 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.382712    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdz5d\" (UniqueName: \"kubernetes.io/projected/0bc8396a-f6c2-4385-ac55-19e2c982772a-kube-api-access-pdz5d\") pod \"0bc8396a-f6c2-4385-ac55-19e2c982772a\" (UID: \"0bc8396a-f6c2-4385-ac55-19e2c982772a\") "
Sep 18 15:57:50.382901 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.382752    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc8396a-f6c2-4385-ac55-19e2c982772a-plugins-config-volume\") pod \"0bc8396a-f6c2-4385-ac55-19e2c982772a\" (UID: \"0bc8396a-f6c2-4385-ac55-19e2c982772a\") "
Sep 18 15:57:50.383180 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.383149    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc8396a-f6c2-4385-ac55-19e2c982772a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0bc8396a-f6c2-4385-ac55-19e2c982772a" (UID: "0bc8396a-f6c2-4385-ac55-19e2c982772a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 15:57:50.385002 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.384975    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc8396a-f6c2-4385-ac55-19e2c982772a-kube-api-access-pdz5d" (OuterVolumeSpecName: "kube-api-access-pdz5d") pod "0bc8396a-f6c2-4385-ac55-19e2c982772a" (UID: "0bc8396a-f6c2-4385-ac55-19e2c982772a"). InnerVolumeSpecName "kube-api-access-pdz5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:57:50.483962 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.483920    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdz5d\" (UniqueName: \"kubernetes.io/projected/0bc8396a-f6c2-4385-ac55-19e2c982772a-kube-api-access-pdz5d\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:50.483962 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.483958    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bc8396a-f6c2-4385-ac55-19e2c982772a-plugins-config-volume\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:50.538056 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.538025    2561 generic.go:356] "Generic (PLEG): container finished" podID="0bc8396a-f6c2-4385-ac55-19e2c982772a" containerID="aec9978a80b919d6daa5aa4182c2e055c4147826b24e2aa158faef913c283f8e" exitCode=0
Sep 18 15:57:50.538229 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.538065    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"
Sep 18 15:57:50.538229 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.538102    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8" event={"ID":"0bc8396a-f6c2-4385-ac55-19e2c982772a","Type":"ContainerDied","Data":"aec9978a80b919d6daa5aa4182c2e055c4147826b24e2aa158faef913c283f8e"}
Sep 18 15:57:50.538229 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.538165    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8" event={"ID":"0bc8396a-f6c2-4385-ac55-19e2c982772a","Type":"ContainerDied","Data":"a8211e93475583b326aa50f727cd443d598858e6349b9ec38f7fcd073dfe2623"}
Sep 18 15:57:50.538229 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.538188    2561 scope.go:117] "RemoveContainer" containerID="aec9978a80b919d6daa5aa4182c2e055c4147826b24e2aa158faef913c283f8e"
Sep 18 15:57:50.548649 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.548628    2561 scope.go:117] "RemoveContainer" containerID="aec9978a80b919d6daa5aa4182c2e055c4147826b24e2aa158faef913c283f8e"
Sep 18 15:57:50.548978 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:57:50.548956    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec9978a80b919d6daa5aa4182c2e055c4147826b24e2aa158faef913c283f8e\": container with ID starting with aec9978a80b919d6daa5aa4182c2e055c4147826b24e2aa158faef913c283f8e not found: ID does not exist" containerID="aec9978a80b919d6daa5aa4182c2e055c4147826b24e2aa158faef913c283f8e"
Sep 18 15:57:50.549027 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.548986    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec9978a80b919d6daa5aa4182c2e055c4147826b24e2aa158faef913c283f8e"} err="failed to get container status \"aec9978a80b919d6daa5aa4182c2e055c4147826b24e2aa158faef913c283f8e\": rpc error: code = NotFound desc = could not find container \"aec9978a80b919d6daa5aa4182c2e055c4147826b24e2aa158faef913c283f8e\": container with ID starting with aec9978a80b919d6daa5aa4182c2e055c4147826b24e2aa158faef913c283f8e not found: ID does not exist"
Sep 18 15:57:50.561232 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.561194    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"]
Sep 18 15:57:50.567614 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:50.567592    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-998130-55bff55487-nb8x8"]
Sep 18 15:57:52.381337 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:52.381292    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bc8396a-f6c2-4385-ac55-19e2c982772a" path="/var/lib/kubelet/pods/0bc8396a-f6c2-4385-ac55-19e2c982772a/volumes"
Sep 18 15:57:54.136067 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.136026    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv" podUID="44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4" containerName="main" containerID="cri-o://803b02d0a331f8646c025fd5770efd24976c06a450040a74780b327bb9642865" gracePeriod=45
Sep 18 15:57:54.183299 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.183260    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l" podUID="b35a3266-1689-4273-b948-7ac4ab16c5d3" containerName="main" containerID="cri-o://61df23f76366a50e9269534d17397e2982194c305f471545cc25c8b914f8fdff" gracePeriod=45
Sep 18 15:57:54.463592 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.463570    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:57:54.466831 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.466810    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:57:54.563225 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.563194    2561 generic.go:356] "Generic (PLEG): container finished" podID="44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4" containerID="803b02d0a331f8646c025fd5770efd24976c06a450040a74780b327bb9642865" exitCode=0
Sep 18 15:57:54.563405 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.563245    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"
Sep 18 15:57:54.563405 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.563270    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv" event={"ID":"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4","Type":"ContainerDied","Data":"803b02d0a331f8646c025fd5770efd24976c06a450040a74780b327bb9642865"}
Sep 18 15:57:54.563405 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.563311    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv" event={"ID":"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4","Type":"ContainerDied","Data":"bbab6cbf1a0ce59ec6f5a9efd5fc2978fa252df7226e9fdec5e79bb182df094b"}
Sep 18 15:57:54.563405 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.563334    2561 scope.go:117] "RemoveContainer" containerID="803b02d0a331f8646c025fd5770efd24976c06a450040a74780b327bb9642865"
Sep 18 15:57:54.564872 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.564845    2561 generic.go:356] "Generic (PLEG): container finished" podID="b35a3266-1689-4273-b948-7ac4ab16c5d3" containerID="61df23f76366a50e9269534d17397e2982194c305f471545cc25c8b914f8fdff" exitCode=0
Sep 18 15:57:54.564988 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.564878    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l" event={"ID":"b35a3266-1689-4273-b948-7ac4ab16c5d3","Type":"ContainerDied","Data":"61df23f76366a50e9269534d17397e2982194c305f471545cc25c8b914f8fdff"}
Sep 18 15:57:54.564988 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.564894    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"
Sep 18 15:57:54.564988 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.564918    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l" event={"ID":"b35a3266-1689-4273-b948-7ac4ab16c5d3","Type":"ContainerDied","Data":"8bc32c004ea0fdd9c94a3273504f7fa576339b4a3e1a78432b061a960f69c6af"}
Sep 18 15:57:54.573416 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.573392    2561 scope.go:117] "RemoveContainer" containerID="803b02d0a331f8646c025fd5770efd24976c06a450040a74780b327bb9642865"
Sep 18 15:57:54.573713 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:57:54.573654    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803b02d0a331f8646c025fd5770efd24976c06a450040a74780b327bb9642865\": container with ID starting with 803b02d0a331f8646c025fd5770efd24976c06a450040a74780b327bb9642865 not found: ID does not exist" containerID="803b02d0a331f8646c025fd5770efd24976c06a450040a74780b327bb9642865"
Sep 18 15:57:54.573822 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.573723    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803b02d0a331f8646c025fd5770efd24976c06a450040a74780b327bb9642865"} err="failed to get container status \"803b02d0a331f8646c025fd5770efd24976c06a450040a74780b327bb9642865\": rpc error: code = NotFound desc = could not find container \"803b02d0a331f8646c025fd5770efd24976c06a450040a74780b327bb9642865\": container with ID starting with 803b02d0a331f8646c025fd5770efd24976c06a450040a74780b327bb9642865 not found: ID does not exist"
Sep 18 15:57:54.573822 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.573746    2561 scope.go:117] "RemoveContainer" containerID="61df23f76366a50e9269534d17397e2982194c305f471545cc25c8b914f8fdff"
Sep 18 15:57:54.581854 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.581834    2561 scope.go:117] "RemoveContainer" containerID="61df23f76366a50e9269534d17397e2982194c305f471545cc25c8b914f8fdff"
Sep 18 15:57:54.582134 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:57:54.582116    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61df23f76366a50e9269534d17397e2982194c305f471545cc25c8b914f8fdff\": container with ID starting with 61df23f76366a50e9269534d17397e2982194c305f471545cc25c8b914f8fdff not found: ID does not exist" containerID="61df23f76366a50e9269534d17397e2982194c305f471545cc25c8b914f8fdff"
Sep 18 15:57:54.582196 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.582145    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61df23f76366a50e9269534d17397e2982194c305f471545cc25c8b914f8fdff"} err="failed to get container status \"61df23f76366a50e9269534d17397e2982194c305f471545cc25c8b914f8fdff\": rpc error: code = NotFound desc = could not find container \"61df23f76366a50e9269534d17397e2982194c305f471545cc25c8b914f8fdff\": container with ID starting with 61df23f76366a50e9269534d17397e2982194c305f471545cc25c8b914f8fdff not found: ID does not exist"
Sep 18 15:57:54.629219 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629193    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-dshm\") pod \"b35a3266-1689-4273-b948-7ac4ab16c5d3\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") "
Sep 18 15:57:54.629347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629230    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-model-cache\") pod \"b35a3266-1689-4273-b948-7ac4ab16c5d3\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") "
Sep 18 15:57:54.629347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629276    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22dww\" (UniqueName: \"kubernetes.io/projected/b35a3266-1689-4273-b948-7ac4ab16c5d3-kube-api-access-22dww\") pod \"b35a3266-1689-4273-b948-7ac4ab16c5d3\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") "
Sep 18 15:57:54.629347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629319    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-tls-certs\") pod \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") "
Sep 18 15:57:54.629347 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629337    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-dshm\") pod \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") "
Sep 18 15:57:54.629562 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629377    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b35a3266-1689-4273-b948-7ac4ab16c5d3-tls-certs\") pod \"b35a3266-1689-4273-b948-7ac4ab16c5d3\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") "
Sep 18 15:57:54.629562 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629413    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjtmg\" (UniqueName: \"kubernetes.io/projected/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-kube-api-access-kjtmg\") pod \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") "
Sep 18 15:57:54.629562 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629465    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-home\") pod \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") "
Sep 18 15:57:54.629562 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629485    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "b35a3266-1689-4273-b948-7ac4ab16c5d3" (UID: "b35a3266-1689-4273-b948-7ac4ab16c5d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.629562 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629512    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-model-cache\") pod \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") "
Sep 18 15:57:54.629562 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629547    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-tmp-dir\") pod \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\" (UID: \"44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4\") "
Sep 18 15:57:54.629926 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629578    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-home\") pod \"b35a3266-1689-4273-b948-7ac4ab16c5d3\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") "
Sep 18 15:57:54.629926 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629614    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-tmp-dir\") pod \"b35a3266-1689-4273-b948-7ac4ab16c5d3\" (UID: \"b35a3266-1689-4273-b948-7ac4ab16c5d3\") "
Sep 18 15:57:54.629926 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.629912    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-model-cache" (OuterVolumeSpecName: "model-cache") pod "44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4" (UID: "44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.630064 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.630002    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b35a3266-1689-4273-b948-7ac4ab16c5d3" (UID: "b35a3266-1689-4273-b948-7ac4ab16c5d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.630064 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.630035    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4" (UID: "44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.630186 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.630162    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-home" (OuterVolumeSpecName: "home") pod "44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4" (UID: "44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.630275 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.630252    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-home" (OuterVolumeSpecName: "home") pod "b35a3266-1689-4273-b948-7ac4ab16c5d3" (UID: "b35a3266-1689-4273-b948-7ac4ab16c5d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.630431 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.630388    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.630431 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.630413    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.630431 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.630426    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.630728 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.630438    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.630728 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.630450    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.630728 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.630461    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.632199 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.632175    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4" (UID: "44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:57:54.632199 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.632184    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-kube-api-access-kjtmg" (OuterVolumeSpecName: "kube-api-access-kjtmg") pod "44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4" (UID: "44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4"). InnerVolumeSpecName "kube-api-access-kjtmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:57:54.632199 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.632193    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-dshm" (OuterVolumeSpecName: "dshm") pod "44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4" (UID: "44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.632365 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.632124    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-dshm" (OuterVolumeSpecName: "dshm") pod "b35a3266-1689-4273-b948-7ac4ab16c5d3" (UID: "b35a3266-1689-4273-b948-7ac4ab16c5d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:57:54.632365 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.632205    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35a3266-1689-4273-b948-7ac4ab16c5d3-kube-api-access-22dww" (OuterVolumeSpecName: "kube-api-access-22dww") pod "b35a3266-1689-4273-b948-7ac4ab16c5d3" (UID: "b35a3266-1689-4273-b948-7ac4ab16c5d3"). InnerVolumeSpecName "kube-api-access-22dww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:57:54.632365 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.632349    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35a3266-1689-4273-b948-7ac4ab16c5d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b35a3266-1689-4273-b948-7ac4ab16c5d3" (UID: "b35a3266-1689-4273-b948-7ac4ab16c5d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:57:54.731338 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.731281    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b35a3266-1689-4273-b948-7ac4ab16c5d3-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.731338 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.731311    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22dww\" (UniqueName: \"kubernetes.io/projected/b35a3266-1689-4273-b948-7ac4ab16c5d3-kube-api-access-22dww\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.731338 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.731326    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.731338 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.731339    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.731539 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.731352    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b35a3266-1689-4273-b948-7ac4ab16c5d3-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.731539 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.731366    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjtmg\" (UniqueName: \"kubernetes.io/projected/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4-kube-api-access-kjtmg\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:57:54.886983 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.886956    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"]
Sep 18 15:57:54.890611 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.890589    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-998130/distinct-w0-998130-kserve-c475bf765-hbkfv"]
Sep 18 15:57:54.903942 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.903921    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"]
Sep 18 15:57:54.908315 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:54.908294    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-998130/simulator-w0-998130-kserve-869fc5b49f-lc54l"]
Sep 18 15:57:56.379829 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:56.379794    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4" path="/var/lib/kubelet/pods/44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4/volumes"
Sep 18 15:57:56.380250 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:57:56.380236    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b35a3266-1689-4273-b948-7ac4ab16c5d3" path="/var/lib/kubelet/pods/b35a3266-1689-4273-b948-7ac4ab16c5d3/volumes"
Sep 18 15:58:54.524114 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:58:54.524083    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"]
Sep 18 15:58:55.612311 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:58:55.612282    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"]
Sep 18 15:59:08.735876 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.735843    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"]
Sep 18 15:59:08.736421 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.736401    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc8396a-f6c2-4385-ac55-19e2c982772a" containerName="payload-processing"
Sep 18 15:59:08.736421 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.736419    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc8396a-f6c2-4385-ac55-19e2c982772a" containerName="payload-processing"
Sep 18 15:59:08.736601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.736438    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4" containerName="main"
Sep 18 15:59:08.736601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.736444    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4" containerName="main"
Sep 18 15:59:08.736601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.736454    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b35a3266-1689-4273-b948-7ac4ab16c5d3" containerName="main"
Sep 18 15:59:08.736601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.736459    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b35a3266-1689-4273-b948-7ac4ab16c5d3" containerName="main"
Sep 18 15:59:08.736601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.736475    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d7f49e1-a73c-4b4f-a86f-6adc7e31f104" containerName="main"
Sep 18 15:59:08.736601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.736482    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d7f49e1-a73c-4b4f-a86f-6adc7e31f104" containerName="main"
Sep 18 15:59:08.736601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.736558    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="44cbf9bb-7c0f-4e98-81a5-569fac0bbcf4" containerName="main"
Sep 18 15:59:08.736601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.736572    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bc8396a-f6c2-4385-ac55-19e2c982772a" containerName="payload-processing"
Sep 18 15:59:08.736601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.736587    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d7f49e1-a73c-4b4f-a86f-6adc7e31f104" containerName="main"
Sep 18 15:59:08.736601 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.736598    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b35a3266-1689-4273-b948-7ac4ab16c5d3" containerName="main"
Sep 18 15:59:08.740085 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.740068    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.743230 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.743210    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-f14aba\"/\"sub-test-model-f14aba-kserve-self-signed-certs\""
Sep 18 15:59:08.751710 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.751685    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"]
Sep 18 15:59:08.817436 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.817406    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-home\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.817436 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.817438    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-dshm\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.817689 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.817461    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-model-cache\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.817689 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.817602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779aa7ea-3c8c-4725-994b-e3c662629efb-tls-certs\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.817785 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.817716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-tmp-dir\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.817785 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.817745    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5x7r\" (UniqueName: \"kubernetes.io/projected/779aa7ea-3c8c-4725-994b-e3c662629efb-kube-api-access-m5x7r\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.918512 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.918470    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-tmp-dir\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.918512 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.918517    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5x7r\" (UniqueName: \"kubernetes.io/projected/779aa7ea-3c8c-4725-994b-e3c662629efb-kube-api-access-m5x7r\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.918826 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.918552    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-home\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.918826 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.918571    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-dshm\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.918826 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.918592    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-model-cache\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.918826 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.918757    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779aa7ea-3c8c-4725-994b-e3c662629efb-tls-certs\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.919041 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.919005    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-tmp-dir\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.919041 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.919026    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-home\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.919133 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.919078    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-model-cache\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.920929 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.920906    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-dshm\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.921242 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.921223    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779aa7ea-3c8c-4725-994b-e3c662629efb-tls-certs\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:08.927787 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:08.927767    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5x7r\" (UniqueName: \"kubernetes.io/projected/779aa7ea-3c8c-4725-994b-e3c662629efb-kube-api-access-m5x7r\") pod \"sub-test-model-f14aba-kserve-56f855c64f-xbd4j\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") " pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:09.049558 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.049480    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:09.323259 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.323232    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"]
Sep 18 15:59:09.327531 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:59:09.327491    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod779aa7ea_3c8c_4725_994b_e3c662629efb.slice/crio-111313b298a5c808757d604945d6a8db1c0f9ce018927c183e52cf2c8a2d585b WatchSource:0}: Error finding container 111313b298a5c808757d604945d6a8db1c0f9ce018927c183e52cf2c8a2d585b: Status 404 returned error can't find the container with id 111313b298a5c808757d604945d6a8db1c0f9ce018927c183e52cf2c8a2d585b
Sep 18 15:59:09.543417 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.543369    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2" podUID="efccf070-2d1b-4446-9a3b-9dc0e30f8260" containerName="main" containerID="cri-o://fbfd179dcbe40805291533bc994f7f1b3de65829da0d93ff97022590f8ff8cf8" gracePeriod=45
Sep 18 15:59:09.772348 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.772325    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:59:09.830556 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.830525    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkh25\" (UniqueName: \"kubernetes.io/projected/efccf070-2d1b-4446-9a3b-9dc0e30f8260-kube-api-access-dkh25\") pod \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") "
Sep 18 15:59:09.830772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.830584    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-dshm\") pod \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") "
Sep 18 15:59:09.830772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.830657    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-home\") pod \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") "
Sep 18 15:59:09.830772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.830734    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efccf070-2d1b-4446-9a3b-9dc0e30f8260-tls-certs\") pod \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") "
Sep 18 15:59:09.830772 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.830767    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-tmp-dir\") pod \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") "
Sep 18 15:59:09.830994 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.830796    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-model-cache\") pod \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\" (UID: \"efccf070-2d1b-4446-9a3b-9dc0e30f8260\") "
Sep 18 15:59:09.831066 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.831006    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-home" (OuterVolumeSpecName: "home") pod "efccf070-2d1b-4446-9a3b-9dc0e30f8260" (UID: "efccf070-2d1b-4446-9a3b-9dc0e30f8260"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:59:09.831066 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.831024    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "efccf070-2d1b-4446-9a3b-9dc0e30f8260" (UID: "efccf070-2d1b-4446-9a3b-9dc0e30f8260"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:59:09.831165 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.831090    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-model-cache" (OuterVolumeSpecName: "model-cache") pod "efccf070-2d1b-4446-9a3b-9dc0e30f8260" (UID: "efccf070-2d1b-4446-9a3b-9dc0e30f8260"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:59:09.831215 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.831204    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:59:09.831266 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.831222    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:59:09.831266 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.831235    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:59:09.832938 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.832909    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efccf070-2d1b-4446-9a3b-9dc0e30f8260-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "efccf070-2d1b-4446-9a3b-9dc0e30f8260" (UID: "efccf070-2d1b-4446-9a3b-9dc0e30f8260"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:59:09.832938 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.832929    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-dshm" (OuterVolumeSpecName: "dshm") pod "efccf070-2d1b-4446-9a3b-9dc0e30f8260" (UID: "efccf070-2d1b-4446-9a3b-9dc0e30f8260"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:59:09.833085 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.832950    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efccf070-2d1b-4446-9a3b-9dc0e30f8260-kube-api-access-dkh25" (OuterVolumeSpecName: "kube-api-access-dkh25") pod "efccf070-2d1b-4446-9a3b-9dc0e30f8260" (UID: "efccf070-2d1b-4446-9a3b-9dc0e30f8260"). InnerVolumeSpecName "kube-api-access-dkh25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:59:09.931817 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.931787    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/efccf070-2d1b-4446-9a3b-9dc0e30f8260-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:59:09.931939 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.931821    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkh25\" (UniqueName: \"kubernetes.io/projected/efccf070-2d1b-4446-9a3b-9dc0e30f8260-kube-api-access-dkh25\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:59:09.931939 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.931838    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/efccf070-2d1b-4446-9a3b-9dc0e30f8260-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:59:09.958270 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.958238    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j" event={"ID":"779aa7ea-3c8c-4725-994b-e3c662629efb","Type":"ContainerStarted","Data":"056c9b9cd8f93e46b6633959a71283a9e4ebc4bbff441400b5d179ae42a1b157"}
Sep 18 15:59:09.958406 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.958291    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j" event={"ID":"779aa7ea-3c8c-4725-994b-e3c662629efb","Type":"ContainerStarted","Data":"111313b298a5c808757d604945d6a8db1c0f9ce018927c183e52cf2c8a2d585b"}
Sep 18 15:59:09.959504 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.959482    2561 generic.go:356] "Generic (PLEG): container finished" podID="efccf070-2d1b-4446-9a3b-9dc0e30f8260" containerID="fbfd179dcbe40805291533bc994f7f1b3de65829da0d93ff97022590f8ff8cf8" exitCode=0
Sep 18 15:59:09.959615 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.959516    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"
Sep 18 15:59:09.959615 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.959557    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2" event={"ID":"efccf070-2d1b-4446-9a3b-9dc0e30f8260","Type":"ContainerDied","Data":"fbfd179dcbe40805291533bc994f7f1b3de65829da0d93ff97022590f8ff8cf8"}
Sep 18 15:59:09.959717 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.959620    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2" event={"ID":"efccf070-2d1b-4446-9a3b-9dc0e30f8260","Type":"ContainerDied","Data":"f915fbc8370e58466fde9fcf98b8548beb633c01204b2adfc15118edfa248908"}
Sep 18 15:59:09.959717 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.959640    2561 scope.go:117] "RemoveContainer" containerID="fbfd179dcbe40805291533bc994f7f1b3de65829da0d93ff97022590f8ff8cf8"
Sep 18 15:59:09.969233 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.969215    2561 scope.go:117] "RemoveContainer" containerID="fbfd179dcbe40805291533bc994f7f1b3de65829da0d93ff97022590f8ff8cf8"
Sep 18 15:59:09.969502 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:59:09.969485    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbfd179dcbe40805291533bc994f7f1b3de65829da0d93ff97022590f8ff8cf8\": container with ID starting with fbfd179dcbe40805291533bc994f7f1b3de65829da0d93ff97022590f8ff8cf8 not found: ID does not exist" containerID="fbfd179dcbe40805291533bc994f7f1b3de65829da0d93ff97022590f8ff8cf8"
Sep 18 15:59:09.969556 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.969510    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbfd179dcbe40805291533bc994f7f1b3de65829da0d93ff97022590f8ff8cf8"} err="failed to get container status \"fbfd179dcbe40805291533bc994f7f1b3de65829da0d93ff97022590f8ff8cf8\": rpc error: code = NotFound desc = could not find container \"fbfd179dcbe40805291533bc994f7f1b3de65829da0d93ff97022590f8ff8cf8\": container with ID starting with fbfd179dcbe40805291533bc994f7f1b3de65829da0d93ff97022590f8ff8cf8 not found: ID does not exist"
Sep 18 15:59:09.977657 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.977615    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j" podStartSLOduration=1.977603796 podStartE2EDuration="1.977603796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:59:09.976026556 +0000 UTC m=+1493.521317236" watchObservedRunningTime="2026-09-18 15:59:09.977603796 +0000 UTC m=+1493.522894470"
Sep 18 15:59:09.990534 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.990513    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"]
Sep 18 15:59:09.995010 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:09.994988    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f14aba/auth-test-model-f14aba-kserve-5d568b5c47-zxnv2"]
Sep 18 15:59:10.381751 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:10.381654    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efccf070-2d1b-4446-9a3b-9dc0e30f8260" path="/var/lib/kubelet/pods/efccf070-2d1b-4446-9a3b-9dc0e30f8260/volumes"
Sep 18 15:59:10.631968 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:10.631881    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c" podUID="41fea5a4-039b-4977-9d62-e4fef8ffc2d1" containerName="main" containerID="cri-o://7dca3f110f0623a448a75bfb9829e276a8f0260004c28a7e935c1d41a427f808" gracePeriod=45
Sep 18 15:59:10.884267 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:10.884206    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:59:10.967391 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:10.967334    2561 generic.go:356] "Generic (PLEG): container finished" podID="41fea5a4-039b-4977-9d62-e4fef8ffc2d1" containerID="7dca3f110f0623a448a75bfb9829e276a8f0260004c28a7e935c1d41a427f808" exitCode=0
Sep 18 15:59:10.967567 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:10.967394    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"
Sep 18 15:59:10.967567 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:10.967429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c" event={"ID":"41fea5a4-039b-4977-9d62-e4fef8ffc2d1","Type":"ContainerDied","Data":"7dca3f110f0623a448a75bfb9829e276a8f0260004c28a7e935c1d41a427f808"}
Sep 18 15:59:10.967567 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:10.967493    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c" event={"ID":"41fea5a4-039b-4977-9d62-e4fef8ffc2d1","Type":"ContainerDied","Data":"febde80a1b78a36b83451d3e2381cd06094ba0ece92ddbc5d28ea1fa4748729e"}
Sep 18 15:59:10.967567 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:10.967518    2561 scope.go:117] "RemoveContainer" containerID="7dca3f110f0623a448a75bfb9829e276a8f0260004c28a7e935c1d41a427f808"
Sep 18 15:59:10.977207 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:10.977195    2561 scope.go:117] "RemoveContainer" containerID="7dca3f110f0623a448a75bfb9829e276a8f0260004c28a7e935c1d41a427f808"
Sep 18 15:59:10.977453 ip-10-0-133-137 kubenswrapper[2561]: E0918 15:59:10.977432    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dca3f110f0623a448a75bfb9829e276a8f0260004c28a7e935c1d41a427f808\": container with ID starting with 7dca3f110f0623a448a75bfb9829e276a8f0260004c28a7e935c1d41a427f808 not found: ID does not exist" containerID="7dca3f110f0623a448a75bfb9829e276a8f0260004c28a7e935c1d41a427f808"
Sep 18 15:59:10.977545 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:10.977459    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dca3f110f0623a448a75bfb9829e276a8f0260004c28a7e935c1d41a427f808"} err="failed to get container status \"7dca3f110f0623a448a75bfb9829e276a8f0260004c28a7e935c1d41a427f808\": rpc error: code = NotFound desc = could not find container \"7dca3f110f0623a448a75bfb9829e276a8f0260004c28a7e935c1d41a427f808\": container with ID starting with 7dca3f110f0623a448a75bfb9829e276a8f0260004c28a7e935c1d41a427f808 not found: ID does not exist"
Sep 18 15:59:11.008646 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.008621    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"]
Sep 18 15:59:11.009061 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.009047    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41fea5a4-039b-4977-9d62-e4fef8ffc2d1" containerName="main"
Sep 18 15:59:11.009112 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.009065    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="41fea5a4-039b-4977-9d62-e4fef8ffc2d1" containerName="main"
Sep 18 15:59:11.009112 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.009088    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efccf070-2d1b-4446-9a3b-9dc0e30f8260" containerName="main"
Sep 18 15:59:11.009112 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.009094    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="efccf070-2d1b-4446-9a3b-9dc0e30f8260" containerName="main"
Sep 18 15:59:11.009210 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.009153    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="efccf070-2d1b-4446-9a3b-9dc0e30f8260" containerName="main"
Sep 18 15:59:11.009210 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.009164    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="41fea5a4-039b-4977-9d62-e4fef8ffc2d1" containerName="main"
Sep 18 15:59:11.013529 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.013508    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"
Sep 18 15:59:11.016082 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.016061    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-769e22\""
Sep 18 15:59:11.016177 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.016167    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-769e22-dockercfg-6bp5z\""
Sep 18 15:59:11.024949 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.024929    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"]
Sep 18 15:59:11.043147 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.043121    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-tmp-dir\") pod \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") "
Sep 18 15:59:11.043258 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.043169    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrt8n\" (UniqueName: \"kubernetes.io/projected/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-kube-api-access-vrt8n\") pod \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") "
Sep 18 15:59:11.043258 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.043224    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-tls-certs\") pod \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") "
Sep 18 15:59:11.043373 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.043285    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-home\") pod \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") "
Sep 18 15:59:11.043373 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.043362    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-dshm\") pod \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") "
Sep 18 15:59:11.043487 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.043389    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-model-cache\") pod \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\" (UID: \"41fea5a4-039b-4977-9d62-e4fef8ffc2d1\") "
Sep 18 15:59:11.043995 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.043386    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41fea5a4-039b-4977-9d62-e4fef8ffc2d1" (UID: "41fea5a4-039b-4977-9d62-e4fef8ffc2d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:59:11.043995 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.043538    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-home" (OuterVolumeSpecName: "home") pod "41fea5a4-039b-4977-9d62-e4fef8ffc2d1" (UID: "41fea5a4-039b-4977-9d62-e4fef8ffc2d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:59:11.043995 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.043842    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:59:11.043995 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.043860    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:59:11.044418 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.044396    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "41fea5a4-039b-4977-9d62-e4fef8ffc2d1" (UID: "41fea5a4-039b-4977-9d62-e4fef8ffc2d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:59:11.046210 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.046185    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-dshm" (OuterVolumeSpecName: "dshm") pod "41fea5a4-039b-4977-9d62-e4fef8ffc2d1" (UID: "41fea5a4-039b-4977-9d62-e4fef8ffc2d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 15:59:11.046322 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.046201    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-kube-api-access-vrt8n" (OuterVolumeSpecName: "kube-api-access-vrt8n") pod "41fea5a4-039b-4977-9d62-e4fef8ffc2d1" (UID: "41fea5a4-039b-4977-9d62-e4fef8ffc2d1"). InnerVolumeSpecName "kube-api-access-vrt8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 15:59:11.046322 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.046185    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41fea5a4-039b-4977-9d62-e4fef8ffc2d1" (UID: "41fea5a4-039b-4977-9d62-e4fef8ffc2d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 15:59:11.144706 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.144602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwbs2\" (UniqueName: \"kubernetes.io/projected/2234f262-6421-43bc-ba7f-97ef2bcf3959-kube-api-access-jwbs2\") pod \"payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn\" (UID: \"2234f262-6421-43bc-ba7f-97ef2bcf3959\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"
Sep 18 15:59:11.144858 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.144734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2234f262-6421-43bc-ba7f-97ef2bcf3959-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn\" (UID: \"2234f262-6421-43bc-ba7f-97ef2bcf3959\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"
Sep 18 15:59:11.144898 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.144864    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrt8n\" (UniqueName: \"kubernetes.io/projected/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-kube-api-access-vrt8n\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:59:11.144898 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.144876    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:59:11.144898 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.144885    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:59:11.144898 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.144894    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41fea5a4-039b-4977-9d62-e4fef8ffc2d1-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 15:59:11.245384 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.245349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2234f262-6421-43bc-ba7f-97ef2bcf3959-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn\" (UID: \"2234f262-6421-43bc-ba7f-97ef2bcf3959\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"
Sep 18 15:59:11.245540 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.245457    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwbs2\" (UniqueName: \"kubernetes.io/projected/2234f262-6421-43bc-ba7f-97ef2bcf3959-kube-api-access-jwbs2\") pod \"payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn\" (UID: \"2234f262-6421-43bc-ba7f-97ef2bcf3959\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"
Sep 18 15:59:11.246165 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.246147    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2234f262-6421-43bc-ba7f-97ef2bcf3959-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn\" (UID: \"2234f262-6421-43bc-ba7f-97ef2bcf3959\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"
Sep 18 15:59:11.256367 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.256341    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwbs2\" (UniqueName: \"kubernetes.io/projected/2234f262-6421-43bc-ba7f-97ef2bcf3959-kube-api-access-jwbs2\") pod \"payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn\" (UID: \"2234f262-6421-43bc-ba7f-97ef2bcf3959\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"
Sep 18 15:59:11.294607 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.294577    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"]
Sep 18 15:59:11.305251 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.305230    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/auth-test-model-4e81aa-kserve-76df8dd787-hxz6c"]
Sep 18 15:59:11.325164 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.325144    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"
Sep 18 15:59:11.602022 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.601993    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"]
Sep 18 15:59:11.603561 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:59:11.603538    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2234f262_6421_43bc_ba7f_97ef2bcf3959.slice/crio-c0f70bd2db2991b4aa63758de45b7433441fd407861893b9187848100e5e84a5 WatchSource:0}: Error finding container c0f70bd2db2991b4aa63758de45b7433441fd407861893b9187848100e5e84a5: Status 404 returned error can't find the container with id c0f70bd2db2991b4aa63758de45b7433441fd407861893b9187848100e5e84a5
Sep 18 15:59:11.975474 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.975428    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn" event={"ID":"2234f262-6421-43bc-ba7f-97ef2bcf3959","Type":"ContainerStarted","Data":"db22c1e05312533c65b072197c4525bf1440c2858919ac9d3f17f04c615db7e2"}
Sep 18 15:59:11.975942 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.975499    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"
Sep 18 15:59:11.975942 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.975523    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn" event={"ID":"2234f262-6421-43bc-ba7f-97ef2bcf3959","Type":"ContainerStarted","Data":"c0f70bd2db2991b4aa63758de45b7433441fd407861893b9187848100e5e84a5"}
Sep 18 15:59:11.993518 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:11.993469    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn" podStartSLOduration=1.993452722 podStartE2EDuration="1.993452722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:59:11.992216251 +0000 UTC m=+1495.537506926" watchObservedRunningTime="2026-09-18 15:59:11.993452722 +0000 UTC m=+1495.538743396"
Sep 18 15:59:12.382459 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:12.382381    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41fea5a4-039b-4977-9d62-e4fef8ffc2d1" path="/var/lib/kubelet/pods/41fea5a4-039b-4977-9d62-e4fef8ffc2d1/volumes"
Sep 18 15:59:18.431723 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:18.431695    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 15:59:18.435942 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:18.435922    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 15:59:19.050062 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:19.050035    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:19.050251 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:19.050071    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:19.062814 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:19.062784    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:20.029936 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:20.029910    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 15:59:22.983783 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:22.983750    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"
Sep 18 15:59:37.442802 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.442729    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"]
Sep 18 15:59:37.446072 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.446056    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.449172 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.449151    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dc538f-openshift-default-dockercfg-kfnz9\""
Sep 18 15:59:37.464249 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.464225    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"]
Sep 18 15:59:37.500142 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.500114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sb86\" (UniqueName: \"kubernetes.io/projected/ac7d021c-c20b-401c-8136-e802f53bbdab-kube-api-access-2sb86\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.500274 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.500148    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-podinfo\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.500274 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.500168    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-workload-socket\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.500274 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.500242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-envoy\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.500421 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.500280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-data\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.500421 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.500311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7d021c-c20b-401c-8136-e802f53bbdab-istiod-ca-cert\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.500421 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.500341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-token\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.500421 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.500360    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-credential-socket\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.500421 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.500393    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-workload-certs\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.601627 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.601589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sb86\" (UniqueName: \"kubernetes.io/projected/ac7d021c-c20b-401c-8136-e802f53bbdab-kube-api-access-2sb86\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.601835 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.601639    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-podinfo\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.601835 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.601698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-workload-socket\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.601835 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.601754    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-envoy\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.601835 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.601803    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-data\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.602058 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.601857    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7d021c-c20b-401c-8136-e802f53bbdab-istiod-ca-cert\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.602058 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.601903    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-token\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.602058 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.601934    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-credential-socket\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.602248 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.602126    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-workload-certs\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.602386 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.602272    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-data\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.602484 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.602412    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-workload-socket\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.602555 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.602470    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-credential-socket\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.602555 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.602515    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-workload-certs\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.602798 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.602774    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7d021c-c20b-401c-8136-e802f53bbdab-istiod-ca-cert\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.604107 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.604086    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-envoy\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.604449 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.604426    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-podinfo\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.633675 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.633637    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-token\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.645515 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.645484    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sb86\" (UniqueName: \"kubernetes.io/projected/ac7d021c-c20b-401c-8136-e802f53bbdab-kube-api-access-2sb86\") pod \"e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") " pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:37.756285 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:37.756200    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:38.036775 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:38.036741    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"]
Sep 18 15:59:38.037802 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:59:38.037777    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7d021c_c20b_401c_8136_e802f53bbdab.slice/crio-9ba44d9c7909b93148d6e8ef8b99ed9844f178208dae463f66cdeede517bff92 WatchSource:0}: Error finding container 9ba44d9c7909b93148d6e8ef8b99ed9844f178208dae463f66cdeede517bff92: Status 404 returned error can't find the container with id 9ba44d9c7909b93148d6e8ef8b99ed9844f178208dae463f66cdeede517bff92
Sep 18 15:59:38.040501 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:38.040467    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:59:38.040609 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:38.040549    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:59:38.040609 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:38.040591    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 15:59:38.115685 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:38.115629    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm" event={"ID":"ac7d021c-c20b-401c-8136-e802f53bbdab","Type":"ContainerStarted","Data":"f12e9c39b5ae664a7d58b60d067690dc0534e44a75604eb1fd3b3661f7784410"}
Sep 18 15:59:38.115818 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:38.115706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm" event={"ID":"ac7d021c-c20b-401c-8136-e802f53bbdab","Type":"ContainerStarted","Data":"9ba44d9c7909b93148d6e8ef8b99ed9844f178208dae463f66cdeede517bff92"}
Sep 18 15:59:38.141758 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:38.141703    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm" podStartSLOduration=1.141682281 podStartE2EDuration="1.141682281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:59:38.139088348 +0000 UTC m=+1521.684379023" watchObservedRunningTime="2026-09-18 15:59:38.141682281 +0000 UTC m=+1521.686972954"
Sep 18 15:59:38.757326 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:38.757297    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:39.762628 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:39.762598    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:40.127885 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:40.127809    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:40.128873 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:40.128851    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 15:59:44.903775 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:44.903740    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"]
Sep 18 15:59:44.907620 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:44.907603    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:44.911842 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:44.911827    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-dc538f-serving-cert\""
Sep 18 15:59:44.932580 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:44.932554    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"]
Sep 18 15:59:44.979083 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:44.979058    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4403ed2-0687-4252-a2f9-f25512d6dc72-openshift-service-ca\") pod \"maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:44.979250 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:44.979229    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4403ed2-0687-4252-a2f9-f25512d6dc72-maas-api-tls\") pod \"maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:44.979322 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:44.979272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78gn4\" (UniqueName: \"kubernetes.io/projected/d4403ed2-0687-4252-a2f9-f25512d6dc72-kube-api-access-78gn4\") pod \"maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:44.979392 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:44.979376    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4403ed2-0687-4252-a2f9-f25512d6dc72-metrics-certs\") pod \"maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:45.080147 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:45.080111    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4403ed2-0687-4252-a2f9-f25512d6dc72-maas-api-tls\") pod \"maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:45.080147 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:45.080150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78gn4\" (UniqueName: \"kubernetes.io/projected/d4403ed2-0687-4252-a2f9-f25512d6dc72-kube-api-access-78gn4\") pod \"maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:45.080395 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:45.080196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4403ed2-0687-4252-a2f9-f25512d6dc72-metrics-certs\") pod \"maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:45.080395 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:45.080360    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4403ed2-0687-4252-a2f9-f25512d6dc72-openshift-service-ca\") pod \"maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:45.080980 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:45.080958    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4403ed2-0687-4252-a2f9-f25512d6dc72-openshift-service-ca\") pod \"maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:45.082678 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:45.082641    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4403ed2-0687-4252-a2f9-f25512d6dc72-metrics-certs\") pod \"maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:45.082783 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:45.082699    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4403ed2-0687-4252-a2f9-f25512d6dc72-maas-api-tls\") pod \"maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:45.095795 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:45.095773    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78gn4\" (UniqueName: \"kubernetes.io/projected/d4403ed2-0687-4252-a2f9-f25512d6dc72-kube-api-access-78gn4\") pod \"maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:45.219670 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:45.219646    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:45.495234 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:45.495203    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"]
Sep 18 15:59:45.497069 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:59:45.497028    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4403ed2_0687_4252_a2f9_f25512d6dc72.slice/crio-f9657eaff893be40f5bae9fc3ecda3286dd1484d77453fc7878fd09a94bdf587 WatchSource:0}: Error finding container f9657eaff893be40f5bae9fc3ecda3286dd1484d77453fc7878fd09a94bdf587: Status 404 returned error can't find the container with id f9657eaff893be40f5bae9fc3ecda3286dd1484d77453fc7878fd09a94bdf587
Sep 18 15:59:45.498889 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:45.498866    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 15:59:46.162658 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:46.162577    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq" event={"ID":"d4403ed2-0687-4252-a2f9-f25512d6dc72","Type":"ContainerStarted","Data":"d8fea2a62a864c2c65fc4d0cc9478e97c430b154245f2c69234cc345b3d6af93"}
Sep 18 15:59:46.162658 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:46.162632    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq" event={"ID":"d4403ed2-0687-4252-a2f9-f25512d6dc72","Type":"ContainerStarted","Data":"f9657eaff893be40f5bae9fc3ecda3286dd1484d77453fc7878fd09a94bdf587"}
Sep 18 15:59:46.163139 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:46.162675    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:46.184076 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:46.184031    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq" podStartSLOduration=1.830070984 podStartE2EDuration="2.184016826s" totalImagesPullingTime="353.945842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 15:59:46.181864818 +0000 UTC m=+1529.727155492" watchObservedRunningTime="2026-09-18 15:59:46.184016826 +0000 UTC m=+1529.729307508"
Sep 18 15:59:52.172980 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:52.172949    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 15:59:59.124460 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.124401    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"]
Sep 18 15:59:59.128981 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.128960    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.132245 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.132222    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4e81aa\"/\"kube-root-ca.crt\""
Sep 18 15:59:59.133404 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.133381    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4e81aa\"/\"default-dockercfg-cm5c9\""
Sep 18 15:59:59.133515 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.133473    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4e81aa\"/\"sub-test-model-4e81aa-kserve-self-signed-certs\""
Sep 18 15:59:59.133515 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.133496    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4e81aa\"/\"openshift-service-ca.crt\""
Sep 18 15:59:59.142348 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.142325    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"]
Sep 18 15:59:59.223882 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.223843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbrvw\" (UniqueName: \"kubernetes.io/projected/7f5c4461-c479-4be2-b01f-6fdaa226a20a-kube-api-access-nbrvw\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.224064 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.223893    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-tmp-dir\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.224064 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.223960    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-home\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.224064 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.224019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-model-cache\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.224064 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.224058    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5c4461-c479-4be2-b01f-6fdaa226a20a-tls-certs\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.224215 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.224126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-dshm\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.325678 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.325563    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-dshm\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.325678 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.325624    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbrvw\" (UniqueName: \"kubernetes.io/projected/7f5c4461-c479-4be2-b01f-6fdaa226a20a-kube-api-access-nbrvw\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.342261 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.325737    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-tmp-dir\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.342261 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.325786    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-home\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.342261 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.325864    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-model-cache\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.342261 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.325916    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5c4461-c479-4be2-b01f-6fdaa226a20a-tls-certs\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.342261 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.326147    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-tmp-dir\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.342261 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.326209    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-home\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.342261 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.326222    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-model-cache\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.342261 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.327998    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-dshm\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.342261 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.328394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5c4461-c479-4be2-b01f-6fdaa226a20a-tls-certs\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.342261 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.338295    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbrvw\" (UniqueName: \"kubernetes.io/projected/7f5c4461-c479-4be2-b01f-6fdaa226a20a-kube-api-access-nbrvw\") pod \"sub-test-model-4e81aa-kserve-7669498ff5-lj5j9\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.441717 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.441657    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 15:59:59.719587 ip-10-0-133-137 kubenswrapper[2561]: I0918 15:59:59.719555    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"]
Sep 18 15:59:59.720165 ip-10-0-133-137 kubenswrapper[2561]: W0918 15:59:59.720139    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f5c4461_c479_4be2_b01f_6fdaa226a20a.slice/crio-e17121c9dda36c54717e57e81b09273a1e117290880b263c2a11c80b98418d20 WatchSource:0}: Error finding container e17121c9dda36c54717e57e81b09273a1e117290880b263c2a11c80b98418d20: Status 404 returned error can't find the container with id e17121c9dda36c54717e57e81b09273a1e117290880b263c2a11c80b98418d20
Sep 18 16:00:00.177389 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.177350    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl"]
Sep 18 16:00:00.201220 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.201180    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl"]
Sep 18 16:00:00.201372 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.201316    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl"
Sep 18 16:00:00.205283 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.205260    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 16:00:00.212735 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.212711    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl"
Sep 18 16:00:00.215616 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.215601    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg"]
Sep 18 16:00:00.220013 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.219996    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg"
Sep 18 16:00:00.232460 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.232442    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg"
Sep 18 16:00:00.243698 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.243636    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg"]
Sep 18 16:00:00.253043 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.253011    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9" event={"ID":"7f5c4461-c479-4be2-b01f-6fdaa226a20a","Type":"ContainerStarted","Data":"807a4f58f08e0e7bef4eeb577104b80d777b88f98119ae4accb33f8ca9e641e2"}
Sep 18 16:00:00.253228 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.253060    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9" event={"ID":"7f5c4461-c479-4be2-b01f-6fdaa226a20a","Type":"ContainerStarted","Data":"e17121c9dda36c54717e57e81b09273a1e117290880b263c2a11c80b98418d20"}
Sep 18 16:00:00.291380 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.291261    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9" podStartSLOduration=1.291237341 podStartE2EDuration="1.291237341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 15:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:00:00.288964652 +0000 UTC m=+1543.834255328" watchObservedRunningTime="2026-09-18 16:00:00.291237341 +0000 UTC m=+1543.836528015"
Sep 18 16:00:00.508516 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.508477    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl"]
Sep 18 16:00:00.509577 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:00:00.509547    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71594ac3_ab90_4e9c_81e5_59504a52b565.slice/crio-062e9986da0d36c7e93e885f8e5ee78c89bc578d576716da838cf773a0fc1f53 WatchSource:0}: Error finding container 062e9986da0d36c7e93e885f8e5ee78c89bc578d576716da838cf773a0fc1f53: Status 404 returned error can't find the container with id 062e9986da0d36c7e93e885f8e5ee78c89bc578d576716da838cf773a0fc1f53
Sep 18 16:00:00.525225 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:00.525200    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg"]
Sep 18 16:00:00.527610 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:00:00.527586    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8516fefb_041a_41ec_bc6e_ce913b308c2b.slice/crio-a34579c51fc5113a2cc0f9369325fd5bc4bc696e85fd5b9fb8607c56c9cb03dc WatchSource:0}: Error finding container a34579c51fc5113a2cc0f9369325fd5bc4bc696e85fd5b9fb8607c56c9cb03dc: Status 404 returned error can't find the container with id a34579c51fc5113a2cc0f9369325fd5bc4bc696e85fd5b9fb8607c56c9cb03dc
Sep 18 16:00:01.259946 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:01.259915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg" event={"ID":"8516fefb-041a-41ec-bc6e-ce913b308c2b","Type":"ContainerStarted","Data":"a34579c51fc5113a2cc0f9369325fd5bc4bc696e85fd5b9fb8607c56c9cb03dc"}
Sep 18 16:00:01.261109 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:01.261087    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl" event={"ID":"71594ac3-ab90-4e9c-81e5-59504a52b565","Type":"ContainerStarted","Data":"062e9986da0d36c7e93e885f8e5ee78c89bc578d576716da838cf773a0fc1f53"}
Sep 18 16:00:01.262624 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:01.262604    2561 generic.go:356] "Generic (PLEG): container finished" podID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerID="f12e9c39b5ae664a7d58b60d067690dc0534e44a75604eb1fd3b3661f7784410" exitCode=0
Sep 18 16:00:01.262722 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:01.262692    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm" event={"ID":"ac7d021c-c20b-401c-8136-e802f53bbdab","Type":"ContainerDied","Data":"f12e9c39b5ae664a7d58b60d067690dc0534e44a75604eb1fd3b3661f7784410"}
Sep 18 16:00:01.263244 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:01.263230    2561 scope.go:117] "RemoveContainer" containerID="f12e9c39b5ae664a7d58b60d067690dc0534e44a75604eb1fd3b3661f7784410"
Sep 18 16:00:01.264289 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:01.264266    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 16:00:01.264356 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:01.264327    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 16:00:01.264393 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:01.264358    2561 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 18 16:00:02.273001 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:02.272963    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm" event={"ID":"ac7d021c-c20b-401c-8136-e802f53bbdab","Type":"ContainerStarted","Data":"f6d9c7b29e070ac2a7fb094f7274d59d6538297d1a1b2dde30a0dd4e752623b3"}
Sep 18 16:00:02.756739 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:02.756700    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 16:00:02.757273 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:02.757239    2561 patch_prober.go:28] interesting pod/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 18 16:00:02.757414 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:02.757302    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm" podUID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 18 16:00:03.756637 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:03.756598    2561 patch_prober.go:28] interesting pod/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 18 16:00:03.757027 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:03.756655    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm" podUID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 18 16:00:04.287466 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:04.287365    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl" event={"ID":"71594ac3-ab90-4e9c-81e5-59504a52b565","Type":"ContainerStarted","Data":"1a42384c85a56811c1eb3abdeba994ea2dbe673158b5e9f4d9c9e8682e00e514"}
Sep 18 16:00:04.289295 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:04.289263    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg" event={"ID":"8516fefb-041a-41ec-bc6e-ce913b308c2b","Type":"ContainerStarted","Data":"68e24916f8f215087018e55a5de1a78fe1f3f53301908a7d7eb73aa769907a88"}
Sep 18 16:00:04.315587 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:04.315533    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl" podStartSLOduration=1.194573447 podStartE2EDuration="4.315517187s" totalImagesPullingTime="3.12094374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:00:04.313859287 +0000 UTC m=+1547.859149963" watchObservedRunningTime="2026-09-18 16:00:04.315517187 +0000 UTC m=+1547.860807860"
Sep 18 16:00:04.345053 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:04.345004    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg" podStartSLOduration=1.237039714 podStartE2EDuration="4.344986617s" totalImagesPullingTime="3.107946903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:00:04.342807347 +0000 UTC m=+1547.888098023" watchObservedRunningTime="2026-09-18 16:00:04.344986617 +0000 UTC m=+1547.890277291"
Sep 18 16:00:04.756489 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:04.756455    2561 patch_prober.go:28] interesting pod/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 18 16:00:04.756657 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:04.756507    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm" podUID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 18 16:00:05.298863 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:05.298768    2561 generic.go:356] "Generic (PLEG): container finished" podID="8516fefb-041a-41ec-bc6e-ce913b308c2b" containerID="68e24916f8f215087018e55a5de1a78fe1f3f53301908a7d7eb73aa769907a88" exitCode=0
Sep 18 16:00:05.299039 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:05.298894    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg" event={"ID":"8516fefb-041a-41ec-bc6e-ce913b308c2b","Type":"ContainerDied","Data":"68e24916f8f215087018e55a5de1a78fe1f3f53301908a7d7eb73aa769907a88"}
Sep 18 16:00:05.301582 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:05.301550    2561 generic.go:356] "Generic (PLEG): container finished" podID="71594ac3-ab90-4e9c-81e5-59504a52b565" containerID="1a42384c85a56811c1eb3abdeba994ea2dbe673158b5e9f4d9c9e8682e00e514" exitCode=0
Sep 18 16:00:05.301735 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:05.301597    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl" event={"ID":"71594ac3-ab90-4e9c-81e5-59504a52b565","Type":"ContainerDied","Data":"1a42384c85a56811c1eb3abdeba994ea2dbe673158b5e9f4d9c9e8682e00e514"}
Sep 18 16:00:05.756935 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:05.756899    2561 patch_prober.go:28] interesting pod/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 18 16:00:05.757317 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:05.756953    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm" podUID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 18 16:00:06.515018 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:06.514993    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl"
Sep 18 16:00:06.519336 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:06.519315    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg"
Sep 18 16:00:06.757460 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:06.757430    2561 patch_prober.go:28] interesting pod/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Sep 18 16:00:06.757853 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:06.757484    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm" podUID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Sep 18 16:00:07.316984 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:07.316951    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg"
Sep 18 16:00:07.316984 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:07.316974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg" event={"ID":"8516fefb-041a-41ec-bc6e-ce913b308c2b","Type":"ContainerDied","Data":"a34579c51fc5113a2cc0f9369325fd5bc4bc696e85fd5b9fb8607c56c9cb03dc"}
Sep 18 16:00:07.317243 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:07.317031    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a34579c51fc5113a2cc0f9369325fd5bc4bc696e85fd5b9fb8607c56c9cb03dc"
Sep 18 16:00:07.318640 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:07.318620    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl"
Sep 18 16:00:07.318788 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:07.318651    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl" event={"ID":"71594ac3-ab90-4e9c-81e5-59504a52b565","Type":"ContainerDied","Data":"062e9986da0d36c7e93e885f8e5ee78c89bc578d576716da838cf773a0fc1f53"}
Sep 18 16:00:07.318788 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:07.318709    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="062e9986da0d36c7e93e885f8e5ee78c89bc578d576716da838cf773a0fc1f53"
Sep 18 16:00:07.761324 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:07.761288    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 16:00:07.761842 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:07.761682    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 16:00:07.762328 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:07.762301    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 16:00:09.442687 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:09.442630    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 16:00:09.443134 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:09.442703    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 16:00:09.455781 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:09.455756    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 16:00:10.348011 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:10.347984    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 16:00:29.120638 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.120602    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg"]
Sep 18 16:00:29.122852 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.122825    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-769e22-29829120-m6vkg"]
Sep 18 16:00:29.157745 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.157716    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"]
Sep 18 16:00:29.157952 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.157920    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn" podUID="2234f262-6421-43bc-ba7f-97ef2bcf3959" containerName="payload-pre-processing" containerID="cri-o://db22c1e05312533c65b072197c4525bf1440c2858919ac9d3f17f04c615db7e2" gracePeriod=30
Sep 18 16:00:29.415308 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.415286    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"
Sep 18 16:00:29.436549 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.436504    2561 generic.go:356] "Generic (PLEG): container finished" podID="2234f262-6421-43bc-ba7f-97ef2bcf3959" containerID="db22c1e05312533c65b072197c4525bf1440c2858919ac9d3f17f04c615db7e2" exitCode=0
Sep 18 16:00:29.436549 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.436546    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"
Sep 18 16:00:29.436788 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.436594    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn" event={"ID":"2234f262-6421-43bc-ba7f-97ef2bcf3959","Type":"ContainerDied","Data":"db22c1e05312533c65b072197c4525bf1440c2858919ac9d3f17f04c615db7e2"}
Sep 18 16:00:29.436788 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.436682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn" event={"ID":"2234f262-6421-43bc-ba7f-97ef2bcf3959","Type":"ContainerDied","Data":"c0f70bd2db2991b4aa63758de45b7433441fd407861893b9187848100e5e84a5"}
Sep 18 16:00:29.436788 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.436700    2561 scope.go:117] "RemoveContainer" containerID="db22c1e05312533c65b072197c4525bf1440c2858919ac9d3f17f04c615db7e2"
Sep 18 16:00:29.447039 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.447016    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2234f262-6421-43bc-ba7f-97ef2bcf3959-plugins-config-volume\") pod \"2234f262-6421-43bc-ba7f-97ef2bcf3959\" (UID: \"2234f262-6421-43bc-ba7f-97ef2bcf3959\") "
Sep 18 16:00:29.447197 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.447047    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwbs2\" (UniqueName: \"kubernetes.io/projected/2234f262-6421-43bc-ba7f-97ef2bcf3959-kube-api-access-jwbs2\") pod \"2234f262-6421-43bc-ba7f-97ef2bcf3959\" (UID: \"2234f262-6421-43bc-ba7f-97ef2bcf3959\") "
Sep 18 16:00:29.447824 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.447434    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2234f262-6421-43bc-ba7f-97ef2bcf3959-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2234f262-6421-43bc-ba7f-97ef2bcf3959" (UID: "2234f262-6421-43bc-ba7f-97ef2bcf3959"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:00:29.447824 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.447511    2561 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2234f262-6421-43bc-ba7f-97ef2bcf3959-plugins-config-volume\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:00:29.448690 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.448641    2561 scope.go:117] "RemoveContainer" containerID="db22c1e05312533c65b072197c4525bf1440c2858919ac9d3f17f04c615db7e2"
Sep 18 16:00:29.449009 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:00:29.448981    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db22c1e05312533c65b072197c4525bf1440c2858919ac9d3f17f04c615db7e2\": container with ID starting with db22c1e05312533c65b072197c4525bf1440c2858919ac9d3f17f04c615db7e2 not found: ID does not exist" containerID="db22c1e05312533c65b072197c4525bf1440c2858919ac9d3f17f04c615db7e2"
Sep 18 16:00:29.449095 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.449018    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db22c1e05312533c65b072197c4525bf1440c2858919ac9d3f17f04c615db7e2"} err="failed to get container status \"db22c1e05312533c65b072197c4525bf1440c2858919ac9d3f17f04c615db7e2\": rpc error: code = NotFound desc = could not find container \"db22c1e05312533c65b072197c4525bf1440c2858919ac9d3f17f04c615db7e2\": container with ID starting with db22c1e05312533c65b072197c4525bf1440c2858919ac9d3f17f04c615db7e2 not found: ID does not exist"
Sep 18 16:00:29.449851 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.449827    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2234f262-6421-43bc-ba7f-97ef2bcf3959-kube-api-access-jwbs2" (OuterVolumeSpecName: "kube-api-access-jwbs2") pod "2234f262-6421-43bc-ba7f-97ef2bcf3959" (UID: "2234f262-6421-43bc-ba7f-97ef2bcf3959"). InnerVolumeSpecName "kube-api-access-jwbs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:00:29.548557 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.548522    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwbs2\" (UniqueName: \"kubernetes.io/projected/2234f262-6421-43bc-ba7f-97ef2bcf3959-kube-api-access-jwbs2\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:00:29.763140 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.763107    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"]
Sep 18 16:00:29.778082 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:29.778054    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-769e22-5d7dd68886-vs9xn"]
Sep 18 16:00:30.380885 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:30.380852    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2234f262-6421-43bc-ba7f-97ef2bcf3959" path="/var/lib/kubelet/pods/2234f262-6421-43bc-ba7f-97ef2bcf3959/volumes"
Sep 18 16:00:30.381265 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:30.381173    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8516fefb-041a-41ec-bc6e-ce913b308c2b" path="/var/lib/kubelet/pods/8516fefb-041a-41ec-bc6e-ce913b308c2b/volumes"
Sep 18 16:00:52.362946 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.362913    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"]
Sep 18 16:00:52.363496 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.363133    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq" podUID="d4403ed2-0687-4252-a2f9-f25512d6dc72" containerName="maas-api" containerID="cri-o://d8fea2a62a864c2c65fc4d0cc9478e97c430b154245f2c69234cc345b3d6af93" gracePeriod=30
Sep 18 16:00:52.566067 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.566027    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4403ed2-0687-4252-a2f9-f25512d6dc72" containerID="d8fea2a62a864c2c65fc4d0cc9478e97c430b154245f2c69234cc345b3d6af93" exitCode=0
Sep 18 16:00:52.566244 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.566101    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq" event={"ID":"d4403ed2-0687-4252-a2f9-f25512d6dc72","Type":"ContainerDied","Data":"d8fea2a62a864c2c65fc4d0cc9478e97c430b154245f2c69234cc345b3d6af93"}
Sep 18 16:00:52.618384 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.618326    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 16:00:52.786291 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.786261    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4403ed2-0687-4252-a2f9-f25512d6dc72-openshift-service-ca\") pod \"d4403ed2-0687-4252-a2f9-f25512d6dc72\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") "
Sep 18 16:00:52.786482 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.786303    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4403ed2-0687-4252-a2f9-f25512d6dc72-maas-api-tls\") pod \"d4403ed2-0687-4252-a2f9-f25512d6dc72\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") "
Sep 18 16:00:52.786482 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.786340    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78gn4\" (UniqueName: \"kubernetes.io/projected/d4403ed2-0687-4252-a2f9-f25512d6dc72-kube-api-access-78gn4\") pod \"d4403ed2-0687-4252-a2f9-f25512d6dc72\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") "
Sep 18 16:00:52.786482 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.786464    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4403ed2-0687-4252-a2f9-f25512d6dc72-metrics-certs\") pod \"d4403ed2-0687-4252-a2f9-f25512d6dc72\" (UID: \"d4403ed2-0687-4252-a2f9-f25512d6dc72\") "
Sep 18 16:00:52.786657 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.786634    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4403ed2-0687-4252-a2f9-f25512d6dc72-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d4403ed2-0687-4252-a2f9-f25512d6dc72" (UID: "d4403ed2-0687-4252-a2f9-f25512d6dc72"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:00:52.786928 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.786904    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4403ed2-0687-4252-a2f9-f25512d6dc72-openshift-service-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:00:52.789092 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.789051    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4403ed2-0687-4252-a2f9-f25512d6dc72-kube-api-access-78gn4" (OuterVolumeSpecName: "kube-api-access-78gn4") pod "d4403ed2-0687-4252-a2f9-f25512d6dc72" (UID: "d4403ed2-0687-4252-a2f9-f25512d6dc72"). InnerVolumeSpecName "kube-api-access-78gn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:00:52.789238 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.789142    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4403ed2-0687-4252-a2f9-f25512d6dc72-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d4403ed2-0687-4252-a2f9-f25512d6dc72" (UID: "d4403ed2-0687-4252-a2f9-f25512d6dc72"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:00:52.789238 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.789163    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4403ed2-0687-4252-a2f9-f25512d6dc72-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d4403ed2-0687-4252-a2f9-f25512d6dc72" (UID: "d4403ed2-0687-4252-a2f9-f25512d6dc72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:00:52.887591 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.887525    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4403ed2-0687-4252-a2f9-f25512d6dc72-metrics-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:00:52.887591 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.887551    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4403ed2-0687-4252-a2f9-f25512d6dc72-maas-api-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:00:52.887591 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:52.887561    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78gn4\" (UniqueName: \"kubernetes.io/projected/d4403ed2-0687-4252-a2f9-f25512d6dc72-kube-api-access-78gn4\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:00:53.573802 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:53.573777    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"
Sep 18 16:00:53.574213 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:53.573797    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq" event={"ID":"d4403ed2-0687-4252-a2f9-f25512d6dc72","Type":"ContainerDied","Data":"f9657eaff893be40f5bae9fc3ecda3286dd1484d77453fc7878fd09a94bdf587"}
Sep 18 16:00:53.574213 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:53.573859    2561 scope.go:117] "RemoveContainer" containerID="d8fea2a62a864c2c65fc4d0cc9478e97c430b154245f2c69234cc345b3d6af93"
Sep 18 16:00:53.598718 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:53.598676    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"]
Sep 18 16:00:53.607232 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:53.607210    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dc538f-865f5749fd-n2tsq"]
Sep 18 16:00:53.874855 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:53.874769    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"]
Sep 18 16:00:54.381600 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:54.381567    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4403ed2-0687-4252-a2f9-f25512d6dc72" path="/var/lib/kubelet/pods/d4403ed2-0687-4252-a2f9-f25512d6dc72/volumes"
Sep 18 16:00:54.964797 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:54.964767    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"]
Sep 18 16:00:59.208932 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:59.208888    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"]
Sep 18 16:00:59.209360 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:00:59.209109    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm" podUID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerName="istio-proxy" containerID="cri-o://f6d9c7b29e070ac2a7fb094f7274d59d6538297d1a1b2dde30a0dd4e752623b3" gracePeriod=30
Sep 18 16:01:04.474056 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.473999    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 16:01:04.496668 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.496642    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-workload-certs\") pod \"ac7d021c-c20b-401c-8136-e802f53bbdab\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") "
Sep 18 16:01:04.496793 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.496693    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-credential-socket\") pod \"ac7d021c-c20b-401c-8136-e802f53bbdab\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") "
Sep 18 16:01:04.496793 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.496718    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-podinfo\") pod \"ac7d021c-c20b-401c-8136-e802f53bbdab\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") "
Sep 18 16:01:04.496793 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.496733    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-token\") pod \"ac7d021c-c20b-401c-8136-e802f53bbdab\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") "
Sep 18 16:01:04.496793 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.496751    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-data\") pod \"ac7d021c-c20b-401c-8136-e802f53bbdab\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") "
Sep 18 16:01:04.497014 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.496817    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-workload-socket\") pod \"ac7d021c-c20b-401c-8136-e802f53bbdab\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") "
Sep 18 16:01:04.497014 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.496849    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-envoy\") pod \"ac7d021c-c20b-401c-8136-e802f53bbdab\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") "
Sep 18 16:01:04.497014 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.496916    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7d021c-c20b-401c-8136-e802f53bbdab-istiod-ca-cert\") pod \"ac7d021c-c20b-401c-8136-e802f53bbdab\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") "
Sep 18 16:01:04.497014 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.496934    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac7d021c-c20b-401c-8136-e802f53bbdab" (UID: "ac7d021c-c20b-401c-8136-e802f53bbdab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:04.497014 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.496977    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sb86\" (UniqueName: \"kubernetes.io/projected/ac7d021c-c20b-401c-8136-e802f53bbdab-kube-api-access-2sb86\") pod \"ac7d021c-c20b-401c-8136-e802f53bbdab\" (UID: \"ac7d021c-c20b-401c-8136-e802f53bbdab\") "
Sep 18 16:01:04.497277 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.497063    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac7d021c-c20b-401c-8136-e802f53bbdab" (UID: "ac7d021c-c20b-401c-8136-e802f53bbdab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:04.497277 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.497138    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac7d021c-c20b-401c-8136-e802f53bbdab" (UID: "ac7d021c-c20b-401c-8136-e802f53bbdab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:04.497382 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.497305    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7d021c-c20b-401c-8136-e802f53bbdab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac7d021c-c20b-401c-8136-e802f53bbdab" (UID: "ac7d021c-c20b-401c-8136-e802f53bbdab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:01:04.497473 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.497433    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac7d021c-c20b-401c-8136-e802f53bbdab" (UID: "ac7d021c-c20b-401c-8136-e802f53bbdab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:04.497567 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.497540    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac7d021c-c20b-401c-8136-e802f53bbdab-istiod-ca-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:04.497567 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.497560    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-workload-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:04.497690 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.497576    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-credential-socket\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:04.497690 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.497590    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-data\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:04.497690 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.497604    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-workload-socket\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:04.499502 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.499469    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac7d021c-c20b-401c-8136-e802f53bbdab" (UID: "ac7d021c-c20b-401c-8136-e802f53bbdab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:01:04.499594 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.499506    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7d021c-c20b-401c-8136-e802f53bbdab-kube-api-access-2sb86" (OuterVolumeSpecName: "kube-api-access-2sb86") pod "ac7d021c-c20b-401c-8136-e802f53bbdab" (UID: "ac7d021c-c20b-401c-8136-e802f53bbdab"). InnerVolumeSpecName "kube-api-access-2sb86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:01:04.499594 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.499497    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac7d021c-c20b-401c-8136-e802f53bbdab" (UID: "ac7d021c-c20b-401c-8136-e802f53bbdab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:01:04.499594 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.499561    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac7d021c-c20b-401c-8136-e802f53bbdab" (UID: "ac7d021c-c20b-401c-8136-e802f53bbdab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:04.598595 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.598565    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-podinfo\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:04.598595 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.598592    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-token\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:04.598595 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.598601    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac7d021c-c20b-401c-8136-e802f53bbdab-istio-envoy\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:04.598822 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.598609    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sb86\" (UniqueName: \"kubernetes.io/projected/ac7d021c-c20b-401c-8136-e802f53bbdab-kube-api-access-2sb86\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:04.638991 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.638961    2561 generic.go:356] "Generic (PLEG): container finished" podID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerID="f6d9c7b29e070ac2a7fb094f7274d59d6538297d1a1b2dde30a0dd4e752623b3" exitCode=0
Sep 18 16:01:04.639116 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.639009    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"
Sep 18 16:01:04.639116 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.639053    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm" event={"ID":"ac7d021c-c20b-401c-8136-e802f53bbdab","Type":"ContainerDied","Data":"f6d9c7b29e070ac2a7fb094f7274d59d6538297d1a1b2dde30a0dd4e752623b3"}
Sep 18 16:01:04.639116 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.639108    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm" event={"ID":"ac7d021c-c20b-401c-8136-e802f53bbdab","Type":"ContainerDied","Data":"9ba44d9c7909b93148d6e8ef8b99ed9844f178208dae463f66cdeede517bff92"}
Sep 18 16:01:04.639233 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.639132    2561 scope.go:117] "RemoveContainer" containerID="f6d9c7b29e070ac2a7fb094f7274d59d6538297d1a1b2dde30a0dd4e752623b3"
Sep 18 16:01:04.648788 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.648772    2561 scope.go:117] "RemoveContainer" containerID="f12e9c39b5ae664a7d58b60d067690dc0534e44a75604eb1fd3b3661f7784410"
Sep 18 16:01:04.656940 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.656922    2561 scope.go:117] "RemoveContainer" containerID="f6d9c7b29e070ac2a7fb094f7274d59d6538297d1a1b2dde30a0dd4e752623b3"
Sep 18 16:01:04.657201 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:01:04.657182    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d9c7b29e070ac2a7fb094f7274d59d6538297d1a1b2dde30a0dd4e752623b3\": container with ID starting with f6d9c7b29e070ac2a7fb094f7274d59d6538297d1a1b2dde30a0dd4e752623b3 not found: ID does not exist" containerID="f6d9c7b29e070ac2a7fb094f7274d59d6538297d1a1b2dde30a0dd4e752623b3"
Sep 18 16:01:04.657244 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.657210    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d9c7b29e070ac2a7fb094f7274d59d6538297d1a1b2dde30a0dd4e752623b3"} err="failed to get container status \"f6d9c7b29e070ac2a7fb094f7274d59d6538297d1a1b2dde30a0dd4e752623b3\": rpc error: code = NotFound desc = could not find container \"f6d9c7b29e070ac2a7fb094f7274d59d6538297d1a1b2dde30a0dd4e752623b3\": container with ID starting with f6d9c7b29e070ac2a7fb094f7274d59d6538297d1a1b2dde30a0dd4e752623b3 not found: ID does not exist"
Sep 18 16:01:04.657244 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.657227    2561 scope.go:117] "RemoveContainer" containerID="f12e9c39b5ae664a7d58b60d067690dc0534e44a75604eb1fd3b3661f7784410"
Sep 18 16:01:04.657495 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:01:04.657478    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f12e9c39b5ae664a7d58b60d067690dc0534e44a75604eb1fd3b3661f7784410\": container with ID starting with f12e9c39b5ae664a7d58b60d067690dc0534e44a75604eb1fd3b3661f7784410 not found: ID does not exist" containerID="f12e9c39b5ae664a7d58b60d067690dc0534e44a75604eb1fd3b3661f7784410"
Sep 18 16:01:04.657540 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.657500    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f12e9c39b5ae664a7d58b60d067690dc0534e44a75604eb1fd3b3661f7784410"} err="failed to get container status \"f12e9c39b5ae664a7d58b60d067690dc0534e44a75604eb1fd3b3661f7784410\": rpc error: code = NotFound desc = could not find container \"f12e9c39b5ae664a7d58b60d067690dc0534e44a75604eb1fd3b3661f7784410\": container with ID starting with f12e9c39b5ae664a7d58b60d067690dc0534e44a75604eb1fd3b3661f7784410 not found: ID does not exist"
Sep 18 16:01:04.661907 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.661885    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"]
Sep 18 16:01:04.666344 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:04.666323    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dc538f-openshift-default-dff7f7c7b-xr8gm"]
Sep 18 16:01:06.379996 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:06.379968    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac7d021c-c20b-401c-8136-e802f53bbdab" path="/var/lib/kubelet/pods/ac7d021c-c20b-401c-8136-e802f53bbdab/volumes"
Sep 18 16:01:08.894587 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:08.894549    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j" podUID="779aa7ea-3c8c-4725-994b-e3c662629efb" containerName="main" containerID="cri-o://056c9b9cd8f93e46b6633959a71283a9e4ebc4bbff441400b5d179ae42a1b157" gracePeriod=45
Sep 18 16:01:09.148384 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.148323    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 16:01:09.238847 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.238807    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-home\") pod \"779aa7ea-3c8c-4725-994b-e3c662629efb\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") "
Sep 18 16:01:09.238847 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.238851    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-tmp-dir\") pod \"779aa7ea-3c8c-4725-994b-e3c662629efb\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") "
Sep 18 16:01:09.239046 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.238873    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779aa7ea-3c8c-4725-994b-e3c662629efb-tls-certs\") pod \"779aa7ea-3c8c-4725-994b-e3c662629efb\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") "
Sep 18 16:01:09.239046 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.238908    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-dshm\") pod \"779aa7ea-3c8c-4725-994b-e3c662629efb\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") "
Sep 18 16:01:09.239046 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.238971    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-model-cache\") pod \"779aa7ea-3c8c-4725-994b-e3c662629efb\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") "
Sep 18 16:01:09.239046 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.239021    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5x7r\" (UniqueName: \"kubernetes.io/projected/779aa7ea-3c8c-4725-994b-e3c662629efb-kube-api-access-m5x7r\") pod \"779aa7ea-3c8c-4725-994b-e3c662629efb\" (UID: \"779aa7ea-3c8c-4725-994b-e3c662629efb\") "
Sep 18 16:01:09.239236 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.239128    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-home" (OuterVolumeSpecName: "home") pod "779aa7ea-3c8c-4725-994b-e3c662629efb" (UID: "779aa7ea-3c8c-4725-994b-e3c662629efb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:09.239236 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.239138    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "779aa7ea-3c8c-4725-994b-e3c662629efb" (UID: "779aa7ea-3c8c-4725-994b-e3c662629efb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:09.239358 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.239285    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-model-cache" (OuterVolumeSpecName: "model-cache") pod "779aa7ea-3c8c-4725-994b-e3c662629efb" (UID: "779aa7ea-3c8c-4725-994b-e3c662629efb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:09.239420 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.239349    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:09.239420 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.239377    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:09.241128 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.241102    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-dshm" (OuterVolumeSpecName: "dshm") pod "779aa7ea-3c8c-4725-994b-e3c662629efb" (UID: "779aa7ea-3c8c-4725-994b-e3c662629efb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:09.241242 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.241150    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/779aa7ea-3c8c-4725-994b-e3c662629efb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "779aa7ea-3c8c-4725-994b-e3c662629efb" (UID: "779aa7ea-3c8c-4725-994b-e3c662629efb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:01:09.241242 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.241225    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/779aa7ea-3c8c-4725-994b-e3c662629efb-kube-api-access-m5x7r" (OuterVolumeSpecName: "kube-api-access-m5x7r") pod "779aa7ea-3c8c-4725-994b-e3c662629efb" (UID: "779aa7ea-3c8c-4725-994b-e3c662629efb"). InnerVolumeSpecName "kube-api-access-m5x7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:01:09.340627 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.340593    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5x7r\" (UniqueName: \"kubernetes.io/projected/779aa7ea-3c8c-4725-994b-e3c662629efb-kube-api-access-m5x7r\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:09.340627 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.340626    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/779aa7ea-3c8c-4725-994b-e3c662629efb-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:09.340627 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.340635    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:09.340627 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.340643    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/779aa7ea-3c8c-4725-994b-e3c662629efb-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:09.668555 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.668520    2561 generic.go:356] "Generic (PLEG): container finished" podID="779aa7ea-3c8c-4725-994b-e3c662629efb" containerID="056c9b9cd8f93e46b6633959a71283a9e4ebc4bbff441400b5d179ae42a1b157" exitCode=0
Sep 18 16:01:09.668774 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.668589    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"
Sep 18 16:01:09.668774 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.668610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j" event={"ID":"779aa7ea-3c8c-4725-994b-e3c662629efb","Type":"ContainerDied","Data":"056c9b9cd8f93e46b6633959a71283a9e4ebc4bbff441400b5d179ae42a1b157"}
Sep 18 16:01:09.668774 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.668682    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j" event={"ID":"779aa7ea-3c8c-4725-994b-e3c662629efb","Type":"ContainerDied","Data":"111313b298a5c808757d604945d6a8db1c0f9ce018927c183e52cf2c8a2d585b"}
Sep 18 16:01:09.668774 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.668693    2561 scope.go:117] "RemoveContainer" containerID="056c9b9cd8f93e46b6633959a71283a9e4ebc4bbff441400b5d179ae42a1b157"
Sep 18 16:01:09.677955 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.677939    2561 scope.go:117] "RemoveContainer" containerID="056c9b9cd8f93e46b6633959a71283a9e4ebc4bbff441400b5d179ae42a1b157"
Sep 18 16:01:09.678256 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:01:09.678237    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056c9b9cd8f93e46b6633959a71283a9e4ebc4bbff441400b5d179ae42a1b157\": container with ID starting with 056c9b9cd8f93e46b6633959a71283a9e4ebc4bbff441400b5d179ae42a1b157 not found: ID does not exist" containerID="056c9b9cd8f93e46b6633959a71283a9e4ebc4bbff441400b5d179ae42a1b157"
Sep 18 16:01:09.678358 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.678261    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056c9b9cd8f93e46b6633959a71283a9e4ebc4bbff441400b5d179ae42a1b157"} err="failed to get container status \"056c9b9cd8f93e46b6633959a71283a9e4ebc4bbff441400b5d179ae42a1b157\": rpc error: code = NotFound desc = could not find container \"056c9b9cd8f93e46b6633959a71283a9e4ebc4bbff441400b5d179ae42a1b157\": container with ID starting with 056c9b9cd8f93e46b6633959a71283a9e4ebc4bbff441400b5d179ae42a1b157 not found: ID does not exist"
Sep 18 16:01:09.700981 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.700953    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"]
Sep 18 16:01:09.705395 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.705375    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-f14aba/sub-test-model-f14aba-kserve-56f855c64f-xbd4j"]
Sep 18 16:01:09.983981 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:09.983868    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9" podUID="7f5c4461-c479-4be2-b01f-6fdaa226a20a" containerName="main" containerID="cri-o://807a4f58f08e0e7bef4eeb577104b80d777b88f98119ae4accb33f8ca9e641e2" gracePeriod=45
Sep 18 16:01:10.242372 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.242315    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 16:01:10.351405 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.351374    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-dshm\") pod \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") "
Sep 18 16:01:10.351405 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.351410    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-tmp-dir\") pod \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") "
Sep 18 16:01:10.351640 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.351427    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-home\") pod \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") "
Sep 18 16:01:10.351640 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.351457    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbrvw\" (UniqueName: \"kubernetes.io/projected/7f5c4461-c479-4be2-b01f-6fdaa226a20a-kube-api-access-nbrvw\") pod \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") "
Sep 18 16:01:10.351640 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.351498    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5c4461-c479-4be2-b01f-6fdaa226a20a-tls-certs\") pod \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") "
Sep 18 16:01:10.351640 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.351561    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-model-cache\") pod \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\" (UID: \"7f5c4461-c479-4be2-b01f-6fdaa226a20a\") "
Sep 18 16:01:10.351895 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.351736    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f5c4461-c479-4be2-b01f-6fdaa226a20a" (UID: "7f5c4461-c479-4be2-b01f-6fdaa226a20a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:10.351895 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.351747    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-home" (OuterVolumeSpecName: "home") pod "7f5c4461-c479-4be2-b01f-6fdaa226a20a" (UID: "7f5c4461-c479-4be2-b01f-6fdaa226a20a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:10.351965 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.351940    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:10.352016 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.351961    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:10.352062 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.352042    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f5c4461-c479-4be2-b01f-6fdaa226a20a" (UID: "7f5c4461-c479-4be2-b01f-6fdaa226a20a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:10.353674 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.353638    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-dshm" (OuterVolumeSpecName: "dshm") pod "7f5c4461-c479-4be2-b01f-6fdaa226a20a" (UID: "7f5c4461-c479-4be2-b01f-6fdaa226a20a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:10.354064 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.354047    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5c4461-c479-4be2-b01f-6fdaa226a20a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f5c4461-c479-4be2-b01f-6fdaa226a20a" (UID: "7f5c4461-c479-4be2-b01f-6fdaa226a20a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:01:10.354149 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.354061    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5c4461-c479-4be2-b01f-6fdaa226a20a-kube-api-access-nbrvw" (OuterVolumeSpecName: "kube-api-access-nbrvw") pod "7f5c4461-c479-4be2-b01f-6fdaa226a20a" (UID: "7f5c4461-c479-4be2-b01f-6fdaa226a20a"). InnerVolumeSpecName "kube-api-access-nbrvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:01:10.382972 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.382946    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="779aa7ea-3c8c-4725-994b-e3c662629efb" path="/var/lib/kubelet/pods/779aa7ea-3c8c-4725-994b-e3c662629efb/volumes"
Sep 18 16:01:10.420236 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.420210    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"]
Sep 18 16:01:10.420432 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.420396    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr" podUID="a261d245-7fea-4a2c-9cda-05f141288650" containerName="maas-api" containerID="cri-o://c3c13945bff3962deceba89ac2c856fb78804b337961c836402e7e2a1486ed57" gracePeriod=30
Sep 18 16:01:10.452742 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.452723    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:10.452742 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.452744    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f5c4461-c479-4be2-b01f-6fdaa226a20a-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:10.452875 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.452753    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbrvw\" (UniqueName: \"kubernetes.io/projected/7f5c4461-c479-4be2-b01f-6fdaa226a20a-kube-api-access-nbrvw\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:10.452875 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.452761    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f5c4461-c479-4be2-b01f-6fdaa226a20a-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:10.653350 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.653328    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 16:01:10.675823 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.675794    2561 generic.go:356] "Generic (PLEG): container finished" podID="a261d245-7fea-4a2c-9cda-05f141288650" containerID="c3c13945bff3962deceba89ac2c856fb78804b337961c836402e7e2a1486ed57" exitCode=0
Sep 18 16:01:10.675932 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.675884    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"
Sep 18 16:01:10.675932 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.675886    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr" event={"ID":"a261d245-7fea-4a2c-9cda-05f141288650","Type":"ContainerDied","Data":"c3c13945bff3962deceba89ac2c856fb78804b337961c836402e7e2a1486ed57"}
Sep 18 16:01:10.676046 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.675932    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr" event={"ID":"a261d245-7fea-4a2c-9cda-05f141288650","Type":"ContainerDied","Data":"55f865c5b36a2511095d6e5b17cc20360aa00afc600d7a9d3932c92a1a829d9b"}
Sep 18 16:01:10.676046 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.675959    2561 scope.go:117] "RemoveContainer" containerID="c3c13945bff3962deceba89ac2c856fb78804b337961c836402e7e2a1486ed57"
Sep 18 16:01:10.677341 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.677320    2561 generic.go:356] "Generic (PLEG): container finished" podID="7f5c4461-c479-4be2-b01f-6fdaa226a20a" containerID="807a4f58f08e0e7bef4eeb577104b80d777b88f98119ae4accb33f8ca9e641e2" exitCode=0
Sep 18 16:01:10.677450 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.677352    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"
Sep 18 16:01:10.677450 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.677353    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9" event={"ID":"7f5c4461-c479-4be2-b01f-6fdaa226a20a","Type":"ContainerDied","Data":"807a4f58f08e0e7bef4eeb577104b80d777b88f98119ae4accb33f8ca9e641e2"}
Sep 18 16:01:10.677450 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.677382    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9" event={"ID":"7f5c4461-c479-4be2-b01f-6fdaa226a20a","Type":"ContainerDied","Data":"e17121c9dda36c54717e57e81b09273a1e117290880b263c2a11c80b98418d20"}
Sep 18 16:01:10.685201 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.685185    2561 scope.go:117] "RemoveContainer" containerID="c3c13945bff3962deceba89ac2c856fb78804b337961c836402e7e2a1486ed57"
Sep 18 16:01:10.685445 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:01:10.685427    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c13945bff3962deceba89ac2c856fb78804b337961c836402e7e2a1486ed57\": container with ID starting with c3c13945bff3962deceba89ac2c856fb78804b337961c836402e7e2a1486ed57 not found: ID does not exist" containerID="c3c13945bff3962deceba89ac2c856fb78804b337961c836402e7e2a1486ed57"
Sep 18 16:01:10.685484 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.685452    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c13945bff3962deceba89ac2c856fb78804b337961c836402e7e2a1486ed57"} err="failed to get container status \"c3c13945bff3962deceba89ac2c856fb78804b337961c836402e7e2a1486ed57\": rpc error: code = NotFound desc = could not find container \"c3c13945bff3962deceba89ac2c856fb78804b337961c836402e7e2a1486ed57\": container with ID starting with c3c13945bff3962deceba89ac2c856fb78804b337961c836402e7e2a1486ed57 not found: ID does not exist"
Sep 18 16:01:10.685484 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.685466    2561 scope.go:117] "RemoveContainer" containerID="807a4f58f08e0e7bef4eeb577104b80d777b88f98119ae4accb33f8ca9e641e2"
Sep 18 16:01:10.693913 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.693780    2561 scope.go:117] "RemoveContainer" containerID="807a4f58f08e0e7bef4eeb577104b80d777b88f98119ae4accb33f8ca9e641e2"
Sep 18 16:01:10.694172 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:01:10.694094    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807a4f58f08e0e7bef4eeb577104b80d777b88f98119ae4accb33f8ca9e641e2\": container with ID starting with 807a4f58f08e0e7bef4eeb577104b80d777b88f98119ae4accb33f8ca9e641e2 not found: ID does not exist" containerID="807a4f58f08e0e7bef4eeb577104b80d777b88f98119ae4accb33f8ca9e641e2"
Sep 18 16:01:10.694172 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.694145    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807a4f58f08e0e7bef4eeb577104b80d777b88f98119ae4accb33f8ca9e641e2"} err="failed to get container status \"807a4f58f08e0e7bef4eeb577104b80d777b88f98119ae4accb33f8ca9e641e2\": rpc error: code = NotFound desc = could not find container \"807a4f58f08e0e7bef4eeb577104b80d777b88f98119ae4accb33f8ca9e641e2\": container with ID starting with 807a4f58f08e0e7bef4eeb577104b80d777b88f98119ae4accb33f8ca9e641e2 not found: ID does not exist"
Sep 18 16:01:10.752779 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.752745    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"]
Sep 18 16:01:10.755547 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.755521    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/sub-test-model-4e81aa-kserve-7669498ff5-lj5j9"]
Sep 18 16:01:10.755687 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.755573    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a261d245-7fea-4a2c-9cda-05f141288650-openshift-service-ca\") pod \"a261d245-7fea-4a2c-9cda-05f141288650\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") "
Sep 18 16:01:10.755687 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.755629    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-metrics-certs\") pod \"a261d245-7fea-4a2c-9cda-05f141288650\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") "
Sep 18 16:01:10.755687 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.755648    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-maas-api-tls\") pod \"a261d245-7fea-4a2c-9cda-05f141288650\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") "
Sep 18 16:01:10.755848 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.755689    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htgqd\" (UniqueName: \"kubernetes.io/projected/a261d245-7fea-4a2c-9cda-05f141288650-kube-api-access-htgqd\") pod \"a261d245-7fea-4a2c-9cda-05f141288650\" (UID: \"a261d245-7fea-4a2c-9cda-05f141288650\") "
Sep 18 16:01:10.756035 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.756011    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a261d245-7fea-4a2c-9cda-05f141288650-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a261d245-7fea-4a2c-9cda-05f141288650" (UID: "a261d245-7fea-4a2c-9cda-05f141288650"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:01:10.757865 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.757839    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a261d245-7fea-4a2c-9cda-05f141288650" (UID: "a261d245-7fea-4a2c-9cda-05f141288650"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:01:10.757963 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.757849    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a261d245-7fea-4a2c-9cda-05f141288650" (UID: "a261d245-7fea-4a2c-9cda-05f141288650"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:01:10.757963 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.757893    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a261d245-7fea-4a2c-9cda-05f141288650-kube-api-access-htgqd" (OuterVolumeSpecName: "kube-api-access-htgqd") pod "a261d245-7fea-4a2c-9cda-05f141288650" (UID: "a261d245-7fea-4a2c-9cda-05f141288650"). InnerVolumeSpecName "kube-api-access-htgqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:01:10.857082 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.857011    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a261d245-7fea-4a2c-9cda-05f141288650-openshift-service-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:10.857082 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.857038    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-metrics-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:10.857082 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.857048    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a261d245-7fea-4a2c-9cda-05f141288650-maas-api-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:10.857082 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:10.857056    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htgqd\" (UniqueName: \"kubernetes.io/projected/a261d245-7fea-4a2c-9cda-05f141288650-kube-api-access-htgqd\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:11.000614 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:11.000586    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"]
Sep 18 16:01:11.007595 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:11.007570    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-195881-69d7b5d66b-6xdfr"]
Sep 18 16:01:12.379983 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:12.379953    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f5c4461-c479-4be2-b01f-6fdaa226a20a" path="/var/lib/kubelet/pods/7f5c4461-c479-4be2-b01f-6fdaa226a20a/volumes"
Sep 18 16:01:12.380352 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:12.380343    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a261d245-7fea-4a2c-9cda-05f141288650" path="/var/lib/kubelet/pods/a261d245-7fea-4a2c-9cda-05f141288650/volumes"
Sep 18 16:01:27.120652 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.120614    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"]
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121020    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2234f262-6421-43bc-ba7f-97ef2bcf3959" containerName="payload-pre-processing"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121033    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2234f262-6421-43bc-ba7f-97ef2bcf3959" containerName="payload-pre-processing"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121047    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a261d245-7fea-4a2c-9cda-05f141288650" containerName="maas-api"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121052    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a261d245-7fea-4a2c-9cda-05f141288650" containerName="maas-api"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121063    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4403ed2-0687-4252-a2f9-f25512d6dc72" containerName="maas-api"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121068    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4403ed2-0687-4252-a2f9-f25512d6dc72" containerName="maas-api"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121075    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="779aa7ea-3c8c-4725-994b-e3c662629efb" containerName="main"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121080    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="779aa7ea-3c8c-4725-994b-e3c662629efb" containerName="main"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121088    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f5c4461-c479-4be2-b01f-6fdaa226a20a" containerName="main"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121093    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f5c4461-c479-4be2-b01f-6fdaa226a20a" containerName="main"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121100    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerName="istio-proxy"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121105    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerName="istio-proxy"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121110    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerName="istio-proxy"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121115    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerName="istio-proxy"
Sep 18 16:01:27.121112 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121121    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8516fefb-041a-41ec-bc6e-ce913b308c2b" containerName="cleanup"
Sep 18 16:01:27.121568 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121126    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8516fefb-041a-41ec-bc6e-ce913b308c2b" containerName="cleanup"
Sep 18 16:01:27.121568 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121139    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71594ac3-ab90-4e9c-81e5-59504a52b565" containerName="cleanup"
Sep 18 16:01:27.121568 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121144    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="71594ac3-ab90-4e9c-81e5-59504a52b565" containerName="cleanup"
Sep 18 16:01:27.121568 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121211    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerName="istio-proxy"
Sep 18 16:01:27.121568 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121218    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f5c4461-c479-4be2-b01f-6fdaa226a20a" containerName="main"
Sep 18 16:01:27.121568 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121224    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4403ed2-0687-4252-a2f9-f25512d6dc72" containerName="maas-api"
Sep 18 16:01:27.121568 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121230    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="779aa7ea-3c8c-4725-994b-e3c662629efb" containerName="main"
Sep 18 16:01:27.121568 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121236    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac7d021c-c20b-401c-8136-e802f53bbdab" containerName="istio-proxy"
Sep 18 16:01:27.121568 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121243    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2234f262-6421-43bc-ba7f-97ef2bcf3959" containerName="payload-pre-processing"
Sep 18 16:01:27.121568 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121248    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="71594ac3-ab90-4e9c-81e5-59504a52b565" containerName="cleanup"
Sep 18 16:01:27.121568 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121253    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a261d245-7fea-4a2c-9cda-05f141288650" containerName="maas-api"
Sep 18 16:01:27.121568 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.121260    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8516fefb-041a-41ec-bc6e-ce913b308c2b" containerName="cleanup"
Sep 18 16:01:27.125478 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.125460    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.128181 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.128162    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4e81aa\"/\"kube-root-ca.crt\""
Sep 18 16:01:27.129219 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.129198    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4e81aa\"/\"default-dockercfg-cm5c9\""
Sep 18 16:01:27.129323 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.129198    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4e81aa\"/\"openshift-service-ca.crt\""
Sep 18 16:01:27.129323 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.129264    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4e81aa\"/\"rate-test-model-4e81aa-kserve-self-signed-certs\""
Sep 18 16:01:27.133470 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.133449    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"]
Sep 18 16:01:27.213047 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.213015    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj5s8\" (UniqueName: \"kubernetes.io/projected/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-kube-api-access-hj5s8\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.213204 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.213053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-model-cache\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.213204 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.213078    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-home\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.213204 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.213105    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-tmp-dir\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.213204 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.213172    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-tls-certs\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.213204 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.213193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-dshm\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.314207 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.314171    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-tls-certs\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.314345 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.314215    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-dshm\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.314345 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.314288    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj5s8\" (UniqueName: \"kubernetes.io/projected/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-kube-api-access-hj5s8\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.314345 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.314313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-model-cache\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.314345 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.314338    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-home\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.314547 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.314362    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-tmp-dir\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.314802 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.314776    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-model-cache\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.314898 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.314789    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-tmp-dir\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.314898 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.314827    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-home\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.316541 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.316524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-dshm\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.316836 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.316817    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-tls-certs\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.322464 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.322439    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj5s8\" (UniqueName: \"kubernetes.io/projected/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-kube-api-access-hj5s8\") pod \"rate-test-model-4e81aa-kserve-5596c9dd59-gccv2\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") " pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.435892 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.435850    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:27.708898 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.708874    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"]
Sep 18 16:01:27.710872 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:01:27.710841    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d91cffb_96f1_4d6d_a4b2_fa98425e3631.slice/crio-2d5f9267501154b31cbf1665a0e3d225b0fbbea3ece29bc5eedcfbb03a3b5445 WatchSource:0}: Error finding container 2d5f9267501154b31cbf1665a0e3d225b0fbbea3ece29bc5eedcfbb03a3b5445: Status 404 returned error can't find the container with id 2d5f9267501154b31cbf1665a0e3d225b0fbbea3ece29bc5eedcfbb03a3b5445
Sep 18 16:01:27.774030 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.774005    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2" event={"ID":"9d91cffb-96f1-4d6d-a4b2-fa98425e3631","Type":"ContainerStarted","Data":"e890ecad65988d11f6d459bc94266c3af832498a9e012dca54e944e611cad34d"}
Sep 18 16:01:27.774260 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.774041    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2" event={"ID":"9d91cffb-96f1-4d6d-a4b2-fa98425e3631","Type":"ContainerStarted","Data":"2d5f9267501154b31cbf1665a0e3d225b0fbbea3ece29bc5eedcfbb03a3b5445"}
Sep 18 16:01:27.794525 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:27.794482    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2" podStartSLOduration=0.79446832 podStartE2EDuration="794.46832ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:01:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:01:27.793172563 +0000 UTC m=+1631.338463237" watchObservedRunningTime="2026-09-18 16:01:27.79446832 +0000 UTC m=+1631.339758994"
Sep 18 16:01:35.898262 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:35.898231    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"]
Sep 18 16:01:35.898685 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:35.898446    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy" containerID="cri-o://dda14807eaeee411c8a68d85e2f2648b67925c6ae9ee2dc64ccfe4a5946c0afc" gracePeriod=30
Sep 18 16:01:36.599778 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:36.599742    2561 patch_prober.go:28] interesting pod/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 16:01:36.599950 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:36.599797    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:01:37.436275 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:37.436242    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:37.436641 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:37.436288    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:37.448269 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:37.448249    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:37.838722 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:37.838623    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:01:41.137697 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.137674    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 16:01:41.242522 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.242491    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq8jg\" (UniqueName: \"kubernetes.io/projected/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-kube-api-access-dq8jg\") pod \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") "
Sep 18 16:01:41.242705 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.242571    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istiod-ca-cert\") pod \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") "
Sep 18 16:01:41.242705 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.242606    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-envoy\") pod \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") "
Sep 18 16:01:41.242705 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.242640    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-token\") pod \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") "
Sep 18 16:01:41.242879 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.242710    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-workload-certs\") pod \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") "
Sep 18 16:01:41.242879 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.242743    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-credential-socket\") pod \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") "
Sep 18 16:01:41.242879 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.242779    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-podinfo\") pod \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") "
Sep 18 16:01:41.242879 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.242804    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-data\") pod \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") "
Sep 18 16:01:41.242879 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.242838    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-workload-socket\") pod \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\" (UID: \"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5\") "
Sep 18 16:01:41.243155 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.243019    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" (UID: "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:01:41.243155 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.243011    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" (UID: "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:41.243155 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.243023    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" (UID: "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:41.243326 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.243207    2561 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istiod-ca-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:41.243326 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.243227    2561 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-workload-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:41.243326 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.243238    2561 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-credential-socket\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:41.243326 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.243262    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" (UID: "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:41.243326 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.243279    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-data" (OuterVolumeSpecName: "istio-data") pod "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" (UID: "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:41.244808 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.244763    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" (UID: "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:01:41.244926 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.244861    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" (UID: "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:01:41.245072 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.245054    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-kube-api-access-dq8jg" (OuterVolumeSpecName: "kube-api-access-dq8jg") pod "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" (UID: "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5"). InnerVolumeSpecName "kube-api-access-dq8jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:01:41.245279 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.245254    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-token" (OuterVolumeSpecName: "istio-token") pod "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" (UID: "0f87a54c-e893-4a9e-9f61-b2922fc8f4e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:01:41.343700 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.343656    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq8jg\" (UniqueName: \"kubernetes.io/projected/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-kube-api-access-dq8jg\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:41.343700 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.343701    2561 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-envoy\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:41.343881 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.343710    2561 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-token\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:41.343881 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.343718    2561 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-podinfo\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:41.343881 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.343725    2561 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-istio-data\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:41.343881 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.343733    2561 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5-workload-socket\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:01:41.849326 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.849288    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerID="dda14807eaeee411c8a68d85e2f2648b67925c6ae9ee2dc64ccfe4a5946c0afc" exitCode=0
Sep 18 16:01:41.849496 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.849336    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"
Sep 18 16:01:41.849496 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.849373    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" event={"ID":"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5","Type":"ContainerDied","Data":"dda14807eaeee411c8a68d85e2f2648b67925c6ae9ee2dc64ccfe4a5946c0afc"}
Sep 18 16:01:41.849496 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.849429    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl" event={"ID":"0f87a54c-e893-4a9e-9f61-b2922fc8f4e5","Type":"ContainerDied","Data":"ac374c4a5fff33e83ee356758ead06ae5beac3f869093c32879c5d7819970bfe"}
Sep 18 16:01:41.849496 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.849451    2561 scope.go:117] "RemoveContainer" containerID="dda14807eaeee411c8a68d85e2f2648b67925c6ae9ee2dc64ccfe4a5946c0afc"
Sep 18 16:01:41.859371 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.859353    2561 scope.go:117] "RemoveContainer" containerID="053345fbe629ecbc7d225f14021eb4e693bdf47ee747dea6a06ec1d7152c8b46"
Sep 18 16:01:41.867421 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.867399    2561 scope.go:117] "RemoveContainer" containerID="dda14807eaeee411c8a68d85e2f2648b67925c6ae9ee2dc64ccfe4a5946c0afc"
Sep 18 16:01:41.867682 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:01:41.867645    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda14807eaeee411c8a68d85e2f2648b67925c6ae9ee2dc64ccfe4a5946c0afc\": container with ID starting with dda14807eaeee411c8a68d85e2f2648b67925c6ae9ee2dc64ccfe4a5946c0afc not found: ID does not exist" containerID="dda14807eaeee411c8a68d85e2f2648b67925c6ae9ee2dc64ccfe4a5946c0afc"
Sep 18 16:01:41.867769 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.867687    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda14807eaeee411c8a68d85e2f2648b67925c6ae9ee2dc64ccfe4a5946c0afc"} err="failed to get container status \"dda14807eaeee411c8a68d85e2f2648b67925c6ae9ee2dc64ccfe4a5946c0afc\": rpc error: code = NotFound desc = could not find container \"dda14807eaeee411c8a68d85e2f2648b67925c6ae9ee2dc64ccfe4a5946c0afc\": container with ID starting with dda14807eaeee411c8a68d85e2f2648b67925c6ae9ee2dc64ccfe4a5946c0afc not found: ID does not exist"
Sep 18 16:01:41.867769 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.867705    2561 scope.go:117] "RemoveContainer" containerID="053345fbe629ecbc7d225f14021eb4e693bdf47ee747dea6a06ec1d7152c8b46"
Sep 18 16:01:41.867963 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:01:41.867940    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053345fbe629ecbc7d225f14021eb4e693bdf47ee747dea6a06ec1d7152c8b46\": container with ID starting with 053345fbe629ecbc7d225f14021eb4e693bdf47ee747dea6a06ec1d7152c8b46 not found: ID does not exist" containerID="053345fbe629ecbc7d225f14021eb4e693bdf47ee747dea6a06ec1d7152c8b46"
Sep 18 16:01:41.868020 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.867968    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053345fbe629ecbc7d225f14021eb4e693bdf47ee747dea6a06ec1d7152c8b46"} err="failed to get container status \"053345fbe629ecbc7d225f14021eb4e693bdf47ee747dea6a06ec1d7152c8b46\": rpc error: code = NotFound desc = could not find container \"053345fbe629ecbc7d225f14021eb4e693bdf47ee747dea6a06ec1d7152c8b46\": container with ID starting with 053345fbe629ecbc7d225f14021eb4e693bdf47ee747dea6a06ec1d7152c8b46 not found: ID does not exist"
Sep 18 16:01:41.892180 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.892148    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"]
Sep 18 16:01:41.900800 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:41.900777    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-655d9c-openshift-default-665fc78f94-2qvxl"]
Sep 18 16:01:42.380496 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:01:42.380465    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" path="/var/lib/kubelet/pods/0f87a54c-e893-4a9e-9f61-b2922fc8f4e5/volumes"
Sep 18 16:02:09.805920 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:09.805884    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"]
Sep 18 16:02:15.867382 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.867345    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"]
Sep 18 16:02:15.867983 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.867960    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy"
Sep 18 16:02:15.868064 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.867987    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy"
Sep 18 16:02:15.868064 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.868032    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy"
Sep 18 16:02:15.868064 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.868042    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy"
Sep 18 16:02:15.868214 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.868140    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy"
Sep 18 16:02:15.868214 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.868158    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f87a54c-e893-4a9e-9f61-b2922fc8f4e5" containerName="istio-proxy"
Sep 18 16:02:15.870060 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.870042    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:15.873595 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.873577    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-90ebe5-serving-cert\""
Sep 18 16:02:15.885578 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.885554    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"]
Sep 18 16:02:15.944710 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.944685    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc668180-ba8a-40bc-a2ef-393c647c9a0a-metrics-certs\") pod \"maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:15.944852 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.944783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc668180-ba8a-40bc-a2ef-393c647c9a0a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:15.944852 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.944805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnn9c\" (UniqueName: \"kubernetes.io/projected/dc668180-ba8a-40bc-a2ef-393c647c9a0a-kube-api-access-fnn9c\") pod \"maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:15.944933 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:15.944867    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc668180-ba8a-40bc-a2ef-393c647c9a0a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:16.045380 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:16.045349    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc668180-ba8a-40bc-a2ef-393c647c9a0a-metrics-certs\") pod \"maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:16.045557 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:16.045422    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc668180-ba8a-40bc-a2ef-393c647c9a0a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:16.045557 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:16.045439    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnn9c\" (UniqueName: \"kubernetes.io/projected/dc668180-ba8a-40bc-a2ef-393c647c9a0a-kube-api-access-fnn9c\") pod \"maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:16.045557 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:16.045462    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc668180-ba8a-40bc-a2ef-393c647c9a0a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:16.046186 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:16.046164    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc668180-ba8a-40bc-a2ef-393c647c9a0a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:16.047956 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:16.047932    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc668180-ba8a-40bc-a2ef-393c647c9a0a-metrics-certs\") pod \"maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:16.048045 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:16.047957    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc668180-ba8a-40bc-a2ef-393c647c9a0a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:16.053618 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:16.053597    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnn9c\" (UniqueName: \"kubernetes.io/projected/dc668180-ba8a-40bc-a2ef-393c647c9a0a-kube-api-access-fnn9c\") pod \"maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:16.180338 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:16.180305    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:16.457778 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:16.457745    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"]
Sep 18 16:02:16.460452 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:02:16.460427    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc668180_ba8a_40bc_a2ef_393c647c9a0a.slice/crio-2e2e2bf75813e4fb083307453a2e81442ac6a83563ca891a025ae28868aa3a5f WatchSource:0}: Error finding container 2e2e2bf75813e4fb083307453a2e81442ac6a83563ca891a025ae28868aa3a5f: Status 404 returned error can't find the container with id 2e2e2bf75813e4fb083307453a2e81442ac6a83563ca891a025ae28868aa3a5f
Sep 18 16:02:17.042005 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:17.041968    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s" event={"ID":"dc668180-ba8a-40bc-a2ef-393c647c9a0a","Type":"ContainerStarted","Data":"2e2e2bf75813e4fb083307453a2e81442ac6a83563ca891a025ae28868aa3a5f"}
Sep 18 16:02:18.048862 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:18.048825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s" event={"ID":"dc668180-ba8a-40bc-a2ef-393c647c9a0a","Type":"ContainerStarted","Data":"f5ef8a648b91b1f34f79206fe2f121718a8643429892cf542776b4e36621b3ae"}
Sep 18 16:02:18.049343 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:18.048954    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:18.067406 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:18.067348    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s" podStartSLOduration=2.2541585570000002 podStartE2EDuration="3.067330459s" totalImagesPullingTime="813.171902ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:02:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:02:18.066322642 +0000 UTC m=+1681.611613316" watchObservedRunningTime="2026-09-18 16:02:18.067330459 +0000 UTC m=+1681.612621134"
Sep 18 16:02:24.058540 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:24.058509    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:02:24.825219 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:24.825182    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2" podUID="9d91cffb-96f1-4d6d-a4b2-fa98425e3631" containerName="main" containerID="cri-o://e890ecad65988d11f6d459bc94266c3af832498a9e012dca54e944e611cad34d" gracePeriod=45
Sep 18 16:02:25.077936 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.077881    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:02:25.083357 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.083325    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d91cffb-96f1-4d6d-a4b2-fa98425e3631" containerID="e890ecad65988d11f6d459bc94266c3af832498a9e012dca54e944e611cad34d" exitCode=0
Sep 18 16:02:25.083494 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.083361    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"
Sep 18 16:02:25.083494 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.083415    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2" event={"ID":"9d91cffb-96f1-4d6d-a4b2-fa98425e3631","Type":"ContainerDied","Data":"e890ecad65988d11f6d459bc94266c3af832498a9e012dca54e944e611cad34d"}
Sep 18 16:02:25.083494 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.083469    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2" event={"ID":"9d91cffb-96f1-4d6d-a4b2-fa98425e3631","Type":"ContainerDied","Data":"2d5f9267501154b31cbf1665a0e3d225b0fbbea3ece29bc5eedcfbb03a3b5445"}
Sep 18 16:02:25.083494 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.083491    2561 scope.go:117] "RemoveContainer" containerID="e890ecad65988d11f6d459bc94266c3af832498a9e012dca54e944e611cad34d"
Sep 18 16:02:25.091466 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.091452    2561 scope.go:117] "RemoveContainer" containerID="e890ecad65988d11f6d459bc94266c3af832498a9e012dca54e944e611cad34d"
Sep 18 16:02:25.091751 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:02:25.091731    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e890ecad65988d11f6d459bc94266c3af832498a9e012dca54e944e611cad34d\": container with ID starting with e890ecad65988d11f6d459bc94266c3af832498a9e012dca54e944e611cad34d not found: ID does not exist" containerID="e890ecad65988d11f6d459bc94266c3af832498a9e012dca54e944e611cad34d"
Sep 18 16:02:25.091812 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.091762    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e890ecad65988d11f6d459bc94266c3af832498a9e012dca54e944e611cad34d"} err="failed to get container status \"e890ecad65988d11f6d459bc94266c3af832498a9e012dca54e944e611cad34d\": rpc error: code = NotFound desc = could not find container \"e890ecad65988d11f6d459bc94266c3af832498a9e012dca54e944e611cad34d\": container with ID starting with e890ecad65988d11f6d459bc94266c3af832498a9e012dca54e944e611cad34d not found: ID does not exist"
Sep 18 16:02:25.233530 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.233495    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-tmp-dir\") pod \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") "
Sep 18 16:02:25.233710 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.233549    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-model-cache\") pod \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") "
Sep 18 16:02:25.233710 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.233568    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-home\") pod \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") "
Sep 18 16:02:25.233710 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.233595    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-dshm\") pod \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") "
Sep 18 16:02:25.233902 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.233722    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-tls-certs\") pod \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") "
Sep 18 16:02:25.233902 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.233755    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj5s8\" (UniqueName: \"kubernetes.io/projected/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-kube-api-access-hj5s8\") pod \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\" (UID: \"9d91cffb-96f1-4d6d-a4b2-fa98425e3631\") "
Sep 18 16:02:25.233902 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.233813    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d91cffb-96f1-4d6d-a4b2-fa98425e3631" (UID: "9d91cffb-96f1-4d6d-a4b2-fa98425e3631"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:02:25.233902 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.233828    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d91cffb-96f1-4d6d-a4b2-fa98425e3631" (UID: "9d91cffb-96f1-4d6d-a4b2-fa98425e3631"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:02:25.233902 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.233898    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-home" (OuterVolumeSpecName: "home") pod "9d91cffb-96f1-4d6d-a4b2-fa98425e3631" (UID: "9d91cffb-96f1-4d6d-a4b2-fa98425e3631"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:02:25.234138 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.234047    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:02:25.234138 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.234060    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:02:25.234138 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.234071    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:02:25.235969 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.235947    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-dshm" (OuterVolumeSpecName: "dshm") pod "9d91cffb-96f1-4d6d-a4b2-fa98425e3631" (UID: "9d91cffb-96f1-4d6d-a4b2-fa98425e3631"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:02:25.235969 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.235956    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-kube-api-access-hj5s8" (OuterVolumeSpecName: "kube-api-access-hj5s8") pod "9d91cffb-96f1-4d6d-a4b2-fa98425e3631" (UID: "9d91cffb-96f1-4d6d-a4b2-fa98425e3631"). InnerVolumeSpecName "kube-api-access-hj5s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:02:25.235969 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.235950    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d91cffb-96f1-4d6d-a4b2-fa98425e3631" (UID: "9d91cffb-96f1-4d6d-a4b2-fa98425e3631"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:02:25.334938 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.334877    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:02:25.334938 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.334899    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:02:25.334938 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.334910    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj5s8\" (UniqueName: \"kubernetes.io/projected/9d91cffb-96f1-4d6d-a4b2-fa98425e3631-kube-api-access-hj5s8\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:02:25.406079 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.406052    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"]
Sep 18 16:02:25.409879 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:25.409857    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4e81aa/rate-test-model-4e81aa-kserve-5596c9dd59-gccv2"]
Sep 18 16:02:26.379992 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:26.379955    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d91cffb-96f1-4d6d-a4b2-fa98425e3631" path="/var/lib/kubelet/pods/9d91cffb-96f1-4d6d-a4b2-fa98425e3631/volumes"
Sep 18 16:02:51.162272 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.162187    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"]
Sep 18 16:02:51.162854 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.162655    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d91cffb-96f1-4d6d-a4b2-fa98425e3631" containerName="main"
Sep 18 16:02:51.162854 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.162689    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d91cffb-96f1-4d6d-a4b2-fa98425e3631" containerName="main"
Sep 18 16:02:51.162854 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.162782    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d91cffb-96f1-4d6d-a4b2-fa98425e3631" containerName="main"
Sep 18 16:02:51.168329 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.168297    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.171757 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.171733    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e41875-serving-cert\""
Sep 18 16:02:51.180165 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.180142    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"]
Sep 18 16:02:51.271637 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.271602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fd2418-ba7a-47b4-9680-158c14ec0576-metrics-certs\") pod \"maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.271637 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.271643    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbz5t\" (UniqueName: \"kubernetes.io/projected/f8fd2418-ba7a-47b4-9680-158c14ec0576-kube-api-access-vbz5t\") pod \"maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.271883 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.271743    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fd2418-ba7a-47b4-9680-158c14ec0576-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.271883 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.271792    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8fd2418-ba7a-47b4-9680-158c14ec0576-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.372742 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.372709    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8fd2418-ba7a-47b4-9680-158c14ec0576-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.372935 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.372770    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fd2418-ba7a-47b4-9680-158c14ec0576-metrics-certs\") pod \"maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.372935 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.372813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbz5t\" (UniqueName: \"kubernetes.io/projected/f8fd2418-ba7a-47b4-9680-158c14ec0576-kube-api-access-vbz5t\") pod \"maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.372935 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.372896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fd2418-ba7a-47b4-9680-158c14ec0576-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.373608 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.373578    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fd2418-ba7a-47b4-9680-158c14ec0576-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.375293 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.375271    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8fd2418-ba7a-47b4-9680-158c14ec0576-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.375428 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.375403    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fd2418-ba7a-47b4-9680-158c14ec0576-metrics-certs\") pod \"maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.381029 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.381009    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbz5t\" (UniqueName: \"kubernetes.io/projected/f8fd2418-ba7a-47b4-9680-158c14ec0576-kube-api-access-vbz5t\") pod \"maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.482570 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.482548    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:51.758531 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:51.758506    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"]
Sep 18 16:02:51.760472 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:02:51.760444    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fd2418_ba7a_47b4_9680_158c14ec0576.slice/crio-e08c1fd08dfd1901bd57449309dfb54c6f9aef2adbb0c0b80306384ef4fb929a WatchSource:0}: Error finding container e08c1fd08dfd1901bd57449309dfb54c6f9aef2adbb0c0b80306384ef4fb929a: Status 404 returned error can't find the container with id e08c1fd08dfd1901bd57449309dfb54c6f9aef2adbb0c0b80306384ef4fb929a
Sep 18 16:02:52.253503 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:52.253471    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89" event={"ID":"f8fd2418-ba7a-47b4-9680-158c14ec0576","Type":"ContainerStarted","Data":"9bfbd6ed28c555178c61adb11510ee64cabe259f74a7c1401b91e5b3771add2f"}
Sep 18 16:02:52.253925 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:52.253528    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89" event={"ID":"f8fd2418-ba7a-47b4-9680-158c14ec0576","Type":"ContainerStarted","Data":"e08c1fd08dfd1901bd57449309dfb54c6f9aef2adbb0c0b80306384ef4fb929a"}
Sep 18 16:02:52.253925 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:52.253605    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:02:52.274153 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:52.274097    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89" podStartSLOduration=0.859424714 podStartE2EDuration="1.274082054s" totalImagesPullingTime="414.65734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:02:52.270987297 +0000 UTC m=+1715.816277989" watchObservedRunningTime="2026-09-18 16:02:52.274082054 +0000 UTC m=+1715.819372751"
Sep 18 16:02:58.264709 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:02:58.264651    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:03:31.320442 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.320407    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"]
Sep 18 16:03:31.325250 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.325230    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.328361 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.328328    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-90ebe5\"/\"openshift-service-ca.crt\""
Sep 18 16:03:31.329520 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.329498    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-90ebe5\"/\"kube-root-ca.crt\""
Sep 18 16:03:31.329636 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.329523    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-90ebe5\"/\"ipp-route-90ebe5-kserve-self-signed-certs\""
Sep 18 16:03:31.329636 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.329593    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-90ebe5\"/\"default-dockercfg-4k7q9\""
Sep 18 16:03:31.338580 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.338558    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"]
Sep 18 16:03:31.434208 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.434178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3f5881-f030-4039-bcce-54dba6206f16-tls-certs\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.434363 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.434218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-home\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.434363 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.434254    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-tmp-dir\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.434363 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.434271    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-dshm\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.434363 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.434335    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7sdv\" (UniqueName: \"kubernetes.io/projected/eb3f5881-f030-4039-bcce-54dba6206f16-kube-api-access-k7sdv\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.434363 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.434357    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-model-cache\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.535767 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.535743    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7sdv\" (UniqueName: \"kubernetes.io/projected/eb3f5881-f030-4039-bcce-54dba6206f16-kube-api-access-k7sdv\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.535767 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.535775    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-model-cache\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.535986 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.535802    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3f5881-f030-4039-bcce-54dba6206f16-tls-certs\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.535986 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.535820    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-home\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.535986 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.535852    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-tmp-dir\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.535986 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.535867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-dshm\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.536219 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.536197    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-model-cache\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.536290 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.536240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-tmp-dir\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.536340 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.536283    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-home\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.538068 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.538049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-dshm\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.538377 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.538359    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3f5881-f030-4039-bcce-54dba6206f16-tls-certs\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.544079 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.544049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7sdv\" (UniqueName: \"kubernetes.io/projected/eb3f5881-f030-4039-bcce-54dba6206f16-kube-api-access-k7sdv\") pod \"ipp-route-90ebe5-kserve-745f8dd699-qjhq6\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") " pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.636513 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.636436    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:31.907114 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:31.907091    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"]
Sep 18 16:03:31.908878 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:03:31.908851    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb3f5881_f030_4039_bcce_54dba6206f16.slice/crio-36bc19454e567c1dc0923c61082e4bcc11c0ae791a188823047c68fe932b9e2a WatchSource:0}: Error finding container 36bc19454e567c1dc0923c61082e4bcc11c0ae791a188823047c68fe932b9e2a: Status 404 returned error can't find the container with id 36bc19454e567c1dc0923c61082e4bcc11c0ae791a188823047c68fe932b9e2a
Sep 18 16:03:32.460757 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:32.460720    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6" event={"ID":"eb3f5881-f030-4039-bcce-54dba6206f16","Type":"ContainerStarted","Data":"612f79e5355c459fda26948139522186eaeb69cf38ec35c3e10818146f2e530d"}
Sep 18 16:03:32.460757 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:32.460764    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6" event={"ID":"eb3f5881-f030-4039-bcce-54dba6206f16","Type":"ContainerStarted","Data":"36bc19454e567c1dc0923c61082e4bcc11c0ae791a188823047c68fe932b9e2a"}
Sep 18 16:03:32.480766 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:32.480723    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6" podStartSLOduration=1.480708383 podStartE2EDuration="1.480708383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:03:32.479167005 +0000 UTC m=+1756.024457679" watchObservedRunningTime="2026-09-18 16:03:32.480708383 +0000 UTC m=+1756.025999060"
Sep 18 16:03:41.637158 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:41.637126    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:41.637158 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:41.637169    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:41.649292 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:41.649270    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:03:42.523205 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:03:42.523178    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:04:18.059413 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.059379    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"]
Sep 18 16:04:18.062903 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.062885    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:18.070989 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:04:18.070962    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-e2e-ipp-cleanup-e2585e-serving-cert\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-e2585e-serving-cert\"" type="*v1.Secret"
Sep 18 16:04:18.071441 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:04:18.071416    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82\" is forbidden: User \"system:node:ip-10-0-133-137.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-133-137.ec2.internal' and this object" podUID="17acf8bc-c79f-4990-a1ff-a8862d176799" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:18.113482 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.113450    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"]
Sep 18 16:04:18.164278 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.164249    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jc6m\" (UniqueName: \"kubernetes.io/projected/17acf8bc-c79f-4990-a1ff-a8862d176799-kube-api-access-9jc6m\") pod \"maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:18.164446 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.164285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17acf8bc-c79f-4990-a1ff-a8862d176799-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:18.164446 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.164305    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17acf8bc-c79f-4990-a1ff-a8862d176799-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:18.164446 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.164421    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17acf8bc-c79f-4990-a1ff-a8862d176799-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:18.265755 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.265724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jc6m\" (UniqueName: \"kubernetes.io/projected/17acf8bc-c79f-4990-a1ff-a8862d176799-kube-api-access-9jc6m\") pod \"maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:18.265928 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.265769    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17acf8bc-c79f-4990-a1ff-a8862d176799-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:18.265928 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.265796    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17acf8bc-c79f-4990-a1ff-a8862d176799-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:18.265928 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.265868    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17acf8bc-c79f-4990-a1ff-a8862d176799-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:18.266548 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.266523    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17acf8bc-c79f-4990-a1ff-a8862d176799-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:18.268393 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.268366    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17acf8bc-c79f-4990-a1ff-a8862d176799-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:18.274863 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.274838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jc6m\" (UniqueName: \"kubernetes.io/projected/17acf8bc-c79f-4990-a1ff-a8862d176799-kube-api-access-9jc6m\") pod \"maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:18.464214 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.464191    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 16:04:18.470708 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:18.470677    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 16:04:19.230283 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:19.230245    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-e2585e-serving-cert\""
Sep 18 16:04:19.239316 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:19.239291    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17acf8bc-c79f-4990-a1ff-a8862d176799-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:19.273287 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:19.273254    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:19.548897 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:19.548870    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"]
Sep 18 16:04:19.550771 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:04:19.550744    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17acf8bc_c79f_4990_a1ff_a8862d176799.slice/crio-06f7a1f0355619301091b12bb59e0a079b9eeac0a0e8745ad22633373021a6eb WatchSource:0}: Error finding container 06f7a1f0355619301091b12bb59e0a079b9eeac0a0e8745ad22633373021a6eb: Status 404 returned error can't find the container with id 06f7a1f0355619301091b12bb59e0a079b9eeac0a0e8745ad22633373021a6eb
Sep 18 16:04:19.700943 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:19.700909    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82" event={"ID":"17acf8bc-c79f-4990-a1ff-a8862d176799","Type":"ContainerStarted","Data":"06f7a1f0355619301091b12bb59e0a079b9eeac0a0e8745ad22633373021a6eb"}
Sep 18 16:04:20.708042 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:20.708005    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82" event={"ID":"17acf8bc-c79f-4990-a1ff-a8862d176799","Type":"ContainerStarted","Data":"30f7d8d72dad9283795cbd41f8caab5d7243abf7d6b0d2515b44a68340506372"}
Sep 18 16:04:20.708449 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:20.708071    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:20.726524 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:20.726480    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82" podStartSLOduration=2.349296987 podStartE2EDuration="2.726467269s" totalImagesPullingTime="377.170282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:04:20.725925761 +0000 UTC m=+1804.271216436" watchObservedRunningTime="2026-09-18 16:04:20.726467269 +0000 UTC m=+1804.271757942"
Sep 18 16:04:26.719072 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:26.719037    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:56.508903 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.508869    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"]
Sep 18 16:04:56.509438 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.509086    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82" podUID="17acf8bc-c79f-4990-a1ff-a8862d176799" containerName="maas-api" containerID="cri-o://30f7d8d72dad9283795cbd41f8caab5d7243abf7d6b0d2515b44a68340506372" gracePeriod=30
Sep 18 16:04:56.759216 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.759159    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:56.836542 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.836509    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17acf8bc-c79f-4990-a1ff-a8862d176799-openshift-service-ca\") pod \"17acf8bc-c79f-4990-a1ff-a8862d176799\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") "
Sep 18 16:04:56.836735 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.836591    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17acf8bc-c79f-4990-a1ff-a8862d176799-metrics-certs\") pod \"17acf8bc-c79f-4990-a1ff-a8862d176799\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") "
Sep 18 16:04:56.836735 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.836653    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17acf8bc-c79f-4990-a1ff-a8862d176799-maas-api-tls\") pod \"17acf8bc-c79f-4990-a1ff-a8862d176799\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") "
Sep 18 16:04:56.836735 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.836700    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jc6m\" (UniqueName: \"kubernetes.io/projected/17acf8bc-c79f-4990-a1ff-a8862d176799-kube-api-access-9jc6m\") pod \"17acf8bc-c79f-4990-a1ff-a8862d176799\" (UID: \"17acf8bc-c79f-4990-a1ff-a8862d176799\") "
Sep 18 16:04:56.836909 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.836887    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17acf8bc-c79f-4990-a1ff-a8862d176799-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17acf8bc-c79f-4990-a1ff-a8862d176799" (UID: "17acf8bc-c79f-4990-a1ff-a8862d176799"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:04:56.838869 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.838842    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17acf8bc-c79f-4990-a1ff-a8862d176799-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17acf8bc-c79f-4990-a1ff-a8862d176799" (UID: "17acf8bc-c79f-4990-a1ff-a8862d176799"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:04:56.838990 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.838968    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17acf8bc-c79f-4990-a1ff-a8862d176799-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "17acf8bc-c79f-4990-a1ff-a8862d176799" (UID: "17acf8bc-c79f-4990-a1ff-a8862d176799"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:04:56.839038 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.838968    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17acf8bc-c79f-4990-a1ff-a8862d176799-kube-api-access-9jc6m" (OuterVolumeSpecName: "kube-api-access-9jc6m") pod "17acf8bc-c79f-4990-a1ff-a8862d176799" (UID: "17acf8bc-c79f-4990-a1ff-a8862d176799"). InnerVolumeSpecName "kube-api-access-9jc6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:04:56.904677 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.904629    2561 generic.go:356] "Generic (PLEG): container finished" podID="17acf8bc-c79f-4990-a1ff-a8862d176799" containerID="30f7d8d72dad9283795cbd41f8caab5d7243abf7d6b0d2515b44a68340506372" exitCode=0
Sep 18 16:04:56.904841 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.904707    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"
Sep 18 16:04:56.904841 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.904703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82" event={"ID":"17acf8bc-c79f-4990-a1ff-a8862d176799","Type":"ContainerDied","Data":"30f7d8d72dad9283795cbd41f8caab5d7243abf7d6b0d2515b44a68340506372"}
Sep 18 16:04:56.904841 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.904822    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82" event={"ID":"17acf8bc-c79f-4990-a1ff-a8862d176799","Type":"ContainerDied","Data":"06f7a1f0355619301091b12bb59e0a079b9eeac0a0e8745ad22633373021a6eb"}
Sep 18 16:04:56.905017 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.904845    2561 scope.go:117] "RemoveContainer" containerID="30f7d8d72dad9283795cbd41f8caab5d7243abf7d6b0d2515b44a68340506372"
Sep 18 16:04:56.915062 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.915042    2561 scope.go:117] "RemoveContainer" containerID="30f7d8d72dad9283795cbd41f8caab5d7243abf7d6b0d2515b44a68340506372"
Sep 18 16:04:56.915378 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:04:56.915358    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f7d8d72dad9283795cbd41f8caab5d7243abf7d6b0d2515b44a68340506372\": container with ID starting with 30f7d8d72dad9283795cbd41f8caab5d7243abf7d6b0d2515b44a68340506372 not found: ID does not exist" containerID="30f7d8d72dad9283795cbd41f8caab5d7243abf7d6b0d2515b44a68340506372"
Sep 18 16:04:56.915433 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.915388    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f7d8d72dad9283795cbd41f8caab5d7243abf7d6b0d2515b44a68340506372"} err="failed to get container status \"30f7d8d72dad9283795cbd41f8caab5d7243abf7d6b0d2515b44a68340506372\": rpc error: code = NotFound desc = could not find container \"30f7d8d72dad9283795cbd41f8caab5d7243abf7d6b0d2515b44a68340506372\": container with ID starting with 30f7d8d72dad9283795cbd41f8caab5d7243abf7d6b0d2515b44a68340506372 not found: ID does not exist"
Sep 18 16:04:56.928570 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.928541    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"]
Sep 18 16:04:56.931422 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.931398    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82"]
Sep 18 16:04:56.937815 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.937789    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17acf8bc-c79f-4990-a1ff-a8862d176799-metrics-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:04:56.937815 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.937810    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17acf8bc-c79f-4990-a1ff-a8862d176799-maas-api-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:04:56.937815 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.937818    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jc6m\" (UniqueName: \"kubernetes.io/projected/17acf8bc-c79f-4990-a1ff-a8862d176799-kube-api-access-9jc6m\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:04:56.937985 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:56.937828    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17acf8bc-c79f-4990-a1ff-a8862d176799-openshift-service-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:04:58.380626 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:58.380588    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17acf8bc-c79f-4990-a1ff-a8862d176799" path="/var/lib/kubelet/pods/17acf8bc-c79f-4990-a1ff-a8862d176799/volumes"
Sep 18 16:04:59.714378 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:04:59.714329    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-e2585e-6c8678cbcd-vxb82" podUID="17acf8bc-c79f-4990-a1ff-a8862d176799" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.81:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:05:22.879035 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:22.879001    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"]
Sep 18 16:05:22.879537 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:22.879225    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s" podUID="dc668180-ba8a-40bc-a2ef-393c647c9a0a" containerName="maas-api" containerID="cri-o://f5ef8a648b91b1f34f79206fe2f121718a8643429892cf542776b4e36621b3ae" gracePeriod=30
Sep 18 16:05:23.051952 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.051925    2561 generic.go:356] "Generic (PLEG): container finished" podID="dc668180-ba8a-40bc-a2ef-393c647c9a0a" containerID="f5ef8a648b91b1f34f79206fe2f121718a8643429892cf542776b4e36621b3ae" exitCode=0
Sep 18 16:05:23.052084 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.051981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s" event={"ID":"dc668180-ba8a-40bc-a2ef-393c647c9a0a","Type":"ContainerDied","Data":"f5ef8a648b91b1f34f79206fe2f121718a8643429892cf542776b4e36621b3ae"}
Sep 18 16:05:23.126948 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.126922    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:05:23.186083 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.186052    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc668180-ba8a-40bc-a2ef-393c647c9a0a-maas-api-tls\") pod \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") "
Sep 18 16:05:23.186275 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.186136    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnn9c\" (UniqueName: \"kubernetes.io/projected/dc668180-ba8a-40bc-a2ef-393c647c9a0a-kube-api-access-fnn9c\") pod \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") "
Sep 18 16:05:23.186275 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.186183    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc668180-ba8a-40bc-a2ef-393c647c9a0a-openshift-service-ca\") pod \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") "
Sep 18 16:05:23.186275 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.186240    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc668180-ba8a-40bc-a2ef-393c647c9a0a-metrics-certs\") pod \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\" (UID: \"dc668180-ba8a-40bc-a2ef-393c647c9a0a\") "
Sep 18 16:05:23.186585 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.186558    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc668180-ba8a-40bc-a2ef-393c647c9a0a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc668180-ba8a-40bc-a2ef-393c647c9a0a" (UID: "dc668180-ba8a-40bc-a2ef-393c647c9a0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:05:23.188336 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.188309    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc668180-ba8a-40bc-a2ef-393c647c9a0a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc668180-ba8a-40bc-a2ef-393c647c9a0a" (UID: "dc668180-ba8a-40bc-a2ef-393c647c9a0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:05:23.188336 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.188323    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc668180-ba8a-40bc-a2ef-393c647c9a0a-kube-api-access-fnn9c" (OuterVolumeSpecName: "kube-api-access-fnn9c") pod "dc668180-ba8a-40bc-a2ef-393c647c9a0a" (UID: "dc668180-ba8a-40bc-a2ef-393c647c9a0a"). InnerVolumeSpecName "kube-api-access-fnn9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:05:23.188472 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.188358    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc668180-ba8a-40bc-a2ef-393c647c9a0a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dc668180-ba8a-40bc-a2ef-393c647c9a0a" (UID: "dc668180-ba8a-40bc-a2ef-393c647c9a0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:05:23.287319 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.287287    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnn9c\" (UniqueName: \"kubernetes.io/projected/dc668180-ba8a-40bc-a2ef-393c647c9a0a-kube-api-access-fnn9c\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:05:23.287319 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.287313    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc668180-ba8a-40bc-a2ef-393c647c9a0a-openshift-service-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:05:23.287497 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.287343    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc668180-ba8a-40bc-a2ef-393c647c9a0a-metrics-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:05:23.287497 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:23.287352    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc668180-ba8a-40bc-a2ef-393c647c9a0a-maas-api-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:05:24.058514 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:24.058483    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"
Sep 18 16:05:24.058514 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:24.058480    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s" event={"ID":"dc668180-ba8a-40bc-a2ef-393c647c9a0a","Type":"ContainerDied","Data":"2e2e2bf75813e4fb083307453a2e81442ac6a83563ca891a025ae28868aa3a5f"}
Sep 18 16:05:24.059021 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:24.058547    2561 scope.go:117] "RemoveContainer" containerID="f5ef8a648b91b1f34f79206fe2f121718a8643429892cf542776b4e36621b3ae"
Sep 18 16:05:24.083691 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:24.083655    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"]
Sep 18 16:05:24.102728 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:24.102707    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-90ebe5-5b86f7857-7s99s"]
Sep 18 16:05:24.380606 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:24.380521    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc668180-ba8a-40bc-a2ef-393c647c9a0a" path="/var/lib/kubelet/pods/dc668180-ba8a-40bc-a2ef-393c647c9a0a/volumes"
Sep 18 16:05:28.644247 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:28.644216    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"]
Sep 18 16:05:43.663461 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.663363    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6" podUID="eb3f5881-f030-4039-bcce-54dba6206f16" containerName="main" containerID="cri-o://612f79e5355c459fda26948139522186eaeb69cf38ec35c3e10818146f2e530d" gracePeriod=45
Sep 18 16:05:43.910339 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.910316    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:05:43.983166 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.983138    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-dshm\") pod \"eb3f5881-f030-4039-bcce-54dba6206f16\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") "
Sep 18 16:05:43.983335 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.983253    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7sdv\" (UniqueName: \"kubernetes.io/projected/eb3f5881-f030-4039-bcce-54dba6206f16-kube-api-access-k7sdv\") pod \"eb3f5881-f030-4039-bcce-54dba6206f16\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") "
Sep 18 16:05:43.983335 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.983278    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-model-cache\") pod \"eb3f5881-f030-4039-bcce-54dba6206f16\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") "
Sep 18 16:05:43.983335 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.983308    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-home\") pod \"eb3f5881-f030-4039-bcce-54dba6206f16\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") "
Sep 18 16:05:43.983335 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.983326    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-tmp-dir\") pod \"eb3f5881-f030-4039-bcce-54dba6206f16\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") "
Sep 18 16:05:43.983551 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.983353    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3f5881-f030-4039-bcce-54dba6206f16-tls-certs\") pod \"eb3f5881-f030-4039-bcce-54dba6206f16\" (UID: \"eb3f5881-f030-4039-bcce-54dba6206f16\") "
Sep 18 16:05:43.983615 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.983566    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb3f5881-f030-4039-bcce-54dba6206f16" (UID: "eb3f5881-f030-4039-bcce-54dba6206f16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:05:43.983790 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.983711    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eb3f5881-f030-4039-bcce-54dba6206f16" (UID: "eb3f5881-f030-4039-bcce-54dba6206f16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:05:43.983790 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.983751    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:05:43.983790 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.983751    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-home" (OuterVolumeSpecName: "home") pod "eb3f5881-f030-4039-bcce-54dba6206f16" (UID: "eb3f5881-f030-4039-bcce-54dba6206f16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:05:43.985612 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.985575    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-dshm" (OuterVolumeSpecName: "dshm") pod "eb3f5881-f030-4039-bcce-54dba6206f16" (UID: "eb3f5881-f030-4039-bcce-54dba6206f16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:05:43.985734 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.985652    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3f5881-f030-4039-bcce-54dba6206f16-kube-api-access-k7sdv" (OuterVolumeSpecName: "kube-api-access-k7sdv") pod "eb3f5881-f030-4039-bcce-54dba6206f16" (UID: "eb3f5881-f030-4039-bcce-54dba6206f16"). InnerVolumeSpecName "kube-api-access-k7sdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:05:43.985734 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:43.985688    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb3f5881-f030-4039-bcce-54dba6206f16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eb3f5881-f030-4039-bcce-54dba6206f16" (UID: "eb3f5881-f030-4039-bcce-54dba6206f16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:05:44.084872 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.084842    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7sdv\" (UniqueName: \"kubernetes.io/projected/eb3f5881-f030-4039-bcce-54dba6206f16-kube-api-access-k7sdv\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:05:44.084872 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.084866    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:05:44.084872 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.084875    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:05:44.085076 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.084885    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eb3f5881-f030-4039-bcce-54dba6206f16-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:05:44.085076 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.084892    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eb3f5881-f030-4039-bcce-54dba6206f16-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:05:44.166755 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.166718    2561 generic.go:356] "Generic (PLEG): container finished" podID="eb3f5881-f030-4039-bcce-54dba6206f16" containerID="612f79e5355c459fda26948139522186eaeb69cf38ec35c3e10818146f2e530d" exitCode=0
Sep 18 16:05:44.166887 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.166776    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"
Sep 18 16:05:44.166887 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.166796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6" event={"ID":"eb3f5881-f030-4039-bcce-54dba6206f16","Type":"ContainerDied","Data":"612f79e5355c459fda26948139522186eaeb69cf38ec35c3e10818146f2e530d"}
Sep 18 16:05:44.166887 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.166844    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6" event={"ID":"eb3f5881-f030-4039-bcce-54dba6206f16","Type":"ContainerDied","Data":"36bc19454e567c1dc0923c61082e4bcc11c0ae791a188823047c68fe932b9e2a"}
Sep 18 16:05:44.166887 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.166865    2561 scope.go:117] "RemoveContainer" containerID="612f79e5355c459fda26948139522186eaeb69cf38ec35c3e10818146f2e530d"
Sep 18 16:05:44.177535 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.177509    2561 scope.go:117] "RemoveContainer" containerID="612f79e5355c459fda26948139522186eaeb69cf38ec35c3e10818146f2e530d"
Sep 18 16:05:44.177946 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:05:44.177925    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612f79e5355c459fda26948139522186eaeb69cf38ec35c3e10818146f2e530d\": container with ID starting with 612f79e5355c459fda26948139522186eaeb69cf38ec35c3e10818146f2e530d not found: ID does not exist" containerID="612f79e5355c459fda26948139522186eaeb69cf38ec35c3e10818146f2e530d"
Sep 18 16:05:44.178045 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.177951    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612f79e5355c459fda26948139522186eaeb69cf38ec35c3e10818146f2e530d"} err="failed to get container status \"612f79e5355c459fda26948139522186eaeb69cf38ec35c3e10818146f2e530d\": rpc error: code = NotFound desc = could not find container \"612f79e5355c459fda26948139522186eaeb69cf38ec35c3e10818146f2e530d\": container with ID starting with 612f79e5355c459fda26948139522186eaeb69cf38ec35c3e10818146f2e530d not found: ID does not exist"
Sep 18 16:05:44.189888 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.189857    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"]
Sep 18 16:05:44.194404 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.194382    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-90ebe5/ipp-route-90ebe5-kserve-745f8dd699-qjhq6"]
Sep 18 16:05:44.207515 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:05:44.207491    2561 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb3f5881_f030_4039_bcce_54dba6206f16.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb3f5881_f030_4039_bcce_54dba6206f16.slice/crio-36bc19454e567c1dc0923c61082e4bcc11c0ae791a188823047c68fe932b9e2a\": RecentStats: unable to find data in memory cache]"
Sep 18 16:05:44.379809 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:05:44.379714    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb3f5881-f030-4039-bcce-54dba6206f16" path="/var/lib/kubelet/pods/eb3f5881-f030-4039-bcce-54dba6206f16/volumes"
Sep 18 16:06:04.103792 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.103757    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"]
Sep 18 16:06:04.104278 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.104050    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89" podUID="f8fd2418-ba7a-47b4-9680-158c14ec0576" containerName="maas-api" containerID="cri-o://9bfbd6ed28c555178c61adb11510ee64cabe259f74a7c1401b91e5b3771add2f" gracePeriod=30
Sep 18 16:06:04.271070 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.271033    2561 generic.go:356] "Generic (PLEG): container finished" podID="f8fd2418-ba7a-47b4-9680-158c14ec0576" containerID="9bfbd6ed28c555178c61adb11510ee64cabe259f74a7c1401b91e5b3771add2f" exitCode=0
Sep 18 16:06:04.271219 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.271106    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89" event={"ID":"f8fd2418-ba7a-47b4-9680-158c14ec0576","Type":"ContainerDied","Data":"9bfbd6ed28c555178c61adb11510ee64cabe259f74a7c1401b91e5b3771add2f"}
Sep 18 16:06:04.361206 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.361144    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:06:04.461985 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.461947    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbz5t\" (UniqueName: \"kubernetes.io/projected/f8fd2418-ba7a-47b4-9680-158c14ec0576-kube-api-access-vbz5t\") pod \"f8fd2418-ba7a-47b4-9680-158c14ec0576\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") "
Sep 18 16:06:04.462169 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.462055    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fd2418-ba7a-47b4-9680-158c14ec0576-openshift-service-ca\") pod \"f8fd2418-ba7a-47b4-9680-158c14ec0576\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") "
Sep 18 16:06:04.462169 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.462148    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8fd2418-ba7a-47b4-9680-158c14ec0576-maas-api-tls\") pod \"f8fd2418-ba7a-47b4-9680-158c14ec0576\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") "
Sep 18 16:06:04.462352 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.462179    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fd2418-ba7a-47b4-9680-158c14ec0576-metrics-certs\") pod \"f8fd2418-ba7a-47b4-9680-158c14ec0576\" (UID: \"f8fd2418-ba7a-47b4-9680-158c14ec0576\") "
Sep 18 16:06:04.462582 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.462553    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8fd2418-ba7a-47b4-9680-158c14ec0576-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f8fd2418-ba7a-47b4-9680-158c14ec0576" (UID: "f8fd2418-ba7a-47b4-9680-158c14ec0576"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:06:04.464627 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.464596    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8fd2418-ba7a-47b4-9680-158c14ec0576-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f8fd2418-ba7a-47b4-9680-158c14ec0576" (UID: "f8fd2418-ba7a-47b4-9680-158c14ec0576"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:06:04.464801 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.464779    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8fd2418-ba7a-47b4-9680-158c14ec0576-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f8fd2418-ba7a-47b4-9680-158c14ec0576" (UID: "f8fd2418-ba7a-47b4-9680-158c14ec0576"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:06:04.465144 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.465113    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8fd2418-ba7a-47b4-9680-158c14ec0576-kube-api-access-vbz5t" (OuterVolumeSpecName: "kube-api-access-vbz5t") pod "f8fd2418-ba7a-47b4-9680-158c14ec0576" (UID: "f8fd2418-ba7a-47b4-9680-158c14ec0576"). InnerVolumeSpecName "kube-api-access-vbz5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:06:04.563856 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.563824    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbz5t\" (UniqueName: \"kubernetes.io/projected/f8fd2418-ba7a-47b4-9680-158c14ec0576-kube-api-access-vbz5t\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:06:04.563856 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.563852    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8fd2418-ba7a-47b4-9680-158c14ec0576-openshift-service-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:06:04.563856 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.563863    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8fd2418-ba7a-47b4-9680-158c14ec0576-maas-api-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:06:04.564081 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:04.563871    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fd2418-ba7a-47b4-9680-158c14ec0576-metrics-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:06:05.278159 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:05.278128    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"
Sep 18 16:06:05.278590 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:05.278163    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89" event={"ID":"f8fd2418-ba7a-47b4-9680-158c14ec0576","Type":"ContainerDied","Data":"e08c1fd08dfd1901bd57449309dfb54c6f9aef2adbb0c0b80306384ef4fb929a"}
Sep 18 16:06:05.278590 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:05.278232    2561 scope.go:117] "RemoveContainer" containerID="9bfbd6ed28c555178c61adb11510ee64cabe259f74a7c1401b91e5b3771add2f"
Sep 18 16:06:05.304706 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:05.304678    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"]
Sep 18 16:06:05.311563 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:05.311540    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e41875-68d877df4d-7kr89"]
Sep 18 16:06:06.380063 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:06.380030    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8fd2418-ba7a-47b4-9680-158c14ec0576" path="/var/lib/kubelet/pods/f8fd2418-ba7a-47b4-9680-158c14ec0576/volumes"
Sep 18 16:06:18.639460 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:18.639428    2561 scope.go:117] "RemoveContainer" containerID="68e24916f8f215087018e55a5de1a78fe1f3f53301908a7d7eb73aa769907a88"
Sep 18 16:06:22.005340 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.005307    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"]
Sep 18 16:06:22.005837 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.005561    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm" podUID="f093215b-e864-4090-9c21-1760112dfaa8" containerName="maas-api" containerID="cri-o://6c99600f928aeee4533897cb90f8c052a1f89837f9cda59a7ccdbc9fb1872dc4" gracePeriod=30
Sep 18 16:06:22.042813 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.042781    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl"]
Sep 18 16:06:22.045655 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.045632    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f14aba-29829120-rbszl"]
Sep 18 16:06:22.253836 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.253814    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 16:06:22.315385 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.315296    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqmhk\" (UniqueName: \"kubernetes.io/projected/f093215b-e864-4090-9c21-1760112dfaa8-kube-api-access-dqmhk\") pod \"f093215b-e864-4090-9c21-1760112dfaa8\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") "
Sep 18 16:06:22.315385 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.315362    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f093215b-e864-4090-9c21-1760112dfaa8-openshift-service-ca\") pod \"f093215b-e864-4090-9c21-1760112dfaa8\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") "
Sep 18 16:06:22.315605 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.315437    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-metrics-certs\") pod \"f093215b-e864-4090-9c21-1760112dfaa8\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") "
Sep 18 16:06:22.315605 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.315506    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-maas-api-tls\") pod \"f093215b-e864-4090-9c21-1760112dfaa8\" (UID: \"f093215b-e864-4090-9c21-1760112dfaa8\") "
Sep 18 16:06:22.315829 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.315799    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f093215b-e864-4090-9c21-1760112dfaa8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f093215b-e864-4090-9c21-1760112dfaa8" (UID: "f093215b-e864-4090-9c21-1760112dfaa8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:06:22.317707 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.317652    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f093215b-e864-4090-9c21-1760112dfaa8-kube-api-access-dqmhk" (OuterVolumeSpecName: "kube-api-access-dqmhk") pod "f093215b-e864-4090-9c21-1760112dfaa8" (UID: "f093215b-e864-4090-9c21-1760112dfaa8"). InnerVolumeSpecName "kube-api-access-dqmhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:06:22.317947 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.317917    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f093215b-e864-4090-9c21-1760112dfaa8" (UID: "f093215b-e864-4090-9c21-1760112dfaa8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:06:22.318044 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.318028    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f093215b-e864-4090-9c21-1760112dfaa8" (UID: "f093215b-e864-4090-9c21-1760112dfaa8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:06:22.364251 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.364224    2561 generic.go:356] "Generic (PLEG): container finished" podID="f093215b-e864-4090-9c21-1760112dfaa8" containerID="6c99600f928aeee4533897cb90f8c052a1f89837f9cda59a7ccdbc9fb1872dc4" exitCode=0
Sep 18 16:06:22.364396 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.364266    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"
Sep 18 16:06:22.364396 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.364297    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm" event={"ID":"f093215b-e864-4090-9c21-1760112dfaa8","Type":"ContainerDied","Data":"6c99600f928aeee4533897cb90f8c052a1f89837f9cda59a7ccdbc9fb1872dc4"}
Sep 18 16:06:22.364396 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.364337    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm" event={"ID":"f093215b-e864-4090-9c21-1760112dfaa8","Type":"ContainerDied","Data":"2d5904380d74bfcb04e491e2f6587d9e2eb822d3f6ca3ee35c2fbb8cb0ba6936"}
Sep 18 16:06:22.364396 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.364358    2561 scope.go:117] "RemoveContainer" containerID="6c99600f928aeee4533897cb90f8c052a1f89837f9cda59a7ccdbc9fb1872dc4"
Sep 18 16:06:22.375927 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.375909    2561 scope.go:117] "RemoveContainer" containerID="6c99600f928aeee4533897cb90f8c052a1f89837f9cda59a7ccdbc9fb1872dc4"
Sep 18 16:06:22.376199 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:06:22.376177    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c99600f928aeee4533897cb90f8c052a1f89837f9cda59a7ccdbc9fb1872dc4\": container with ID starting with 6c99600f928aeee4533897cb90f8c052a1f89837f9cda59a7ccdbc9fb1872dc4 not found: ID does not exist" containerID="6c99600f928aeee4533897cb90f8c052a1f89837f9cda59a7ccdbc9fb1872dc4"
Sep 18 16:06:22.376285 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.376203    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c99600f928aeee4533897cb90f8c052a1f89837f9cda59a7ccdbc9fb1872dc4"} err="failed to get container status \"6c99600f928aeee4533897cb90f8c052a1f89837f9cda59a7ccdbc9fb1872dc4\": rpc error: code = NotFound desc = could not find container \"6c99600f928aeee4533897cb90f8c052a1f89837f9cda59a7ccdbc9fb1872dc4\": container with ID starting with 6c99600f928aeee4533897cb90f8c052a1f89837f9cda59a7ccdbc9fb1872dc4 not found: ID does not exist"
Sep 18 16:06:22.380082 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.380060    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71594ac3-ab90-4e9c-81e5-59504a52b565" path="/var/lib/kubelet/pods/71594ac3-ab90-4e9c-81e5-59504a52b565/volumes"
Sep 18 16:06:22.414203 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.414177    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"]
Sep 18 16:06:22.416224 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.416205    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f093215b-e864-4090-9c21-1760112dfaa8-openshift-service-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:06:22.416308 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.416225    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-metrics-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:06:22.416308 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.416236    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f093215b-e864-4090-9c21-1760112dfaa8-maas-api-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:06:22.416308 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.416244    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqmhk\" (UniqueName: \"kubernetes.io/projected/f093215b-e864-4090-9c21-1760112dfaa8-kube-api-access-dqmhk\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:06:22.419601 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:22.419580    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f14aba-5bdd47cd78-hxwtm"]
Sep 18 16:06:24.380307 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:24.380272    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f093215b-e864-4090-9c21-1760112dfaa8" path="/var/lib/kubelet/pods/f093215b-e864-4090-9c21-1760112dfaa8/volumes"
Sep 18 16:06:29.298927 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.298884    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm"]
Sep 18 16:06:29.299487 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299468    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3f5881-f030-4039-bcce-54dba6206f16" containerName="main"
Sep 18 16:06:29.299543 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299491    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3f5881-f030-4039-bcce-54dba6206f16" containerName="main"
Sep 18 16:06:29.299543 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299507    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f093215b-e864-4090-9c21-1760112dfaa8" containerName="maas-api"
Sep 18 16:06:29.299543 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299517    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f093215b-e864-4090-9c21-1760112dfaa8" containerName="maas-api"
Sep 18 16:06:29.299543 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299534    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc668180-ba8a-40bc-a2ef-393c647c9a0a" containerName="maas-api"
Sep 18 16:06:29.299710 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299544    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc668180-ba8a-40bc-a2ef-393c647c9a0a" containerName="maas-api"
Sep 18 16:06:29.299710 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299557    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8fd2418-ba7a-47b4-9680-158c14ec0576" containerName="maas-api"
Sep 18 16:06:29.299710 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299565    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8fd2418-ba7a-47b4-9680-158c14ec0576" containerName="maas-api"
Sep 18 16:06:29.299710 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299583    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17acf8bc-c79f-4990-a1ff-a8862d176799" containerName="maas-api"
Sep 18 16:06:29.299710 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299591    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="17acf8bc-c79f-4990-a1ff-a8862d176799" containerName="maas-api"
Sep 18 16:06:29.299888 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299711    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f093215b-e864-4090-9c21-1760112dfaa8" containerName="maas-api"
Sep 18 16:06:29.299888 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299724    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc668180-ba8a-40bc-a2ef-393c647c9a0a" containerName="maas-api"
Sep 18 16:06:29.299888 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299736    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb3f5881-f030-4039-bcce-54dba6206f16" containerName="main"
Sep 18 16:06:29.299888 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299745    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="17acf8bc-c79f-4990-a1ff-a8862d176799" containerName="maas-api"
Sep 18 16:06:29.299888 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.299755    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8fd2418-ba7a-47b4-9680-158c14ec0576" containerName="maas-api"
Sep 18 16:06:29.304600 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.304582    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm"
Sep 18 16:06:29.307505 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.307342    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8hg46\""
Sep 18 16:06:29.310016 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.309990    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm"]
Sep 18 16:06:29.375138 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.375103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5417439-8eab-4013-b636-0162d82363c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm\" (UID: \"d5417439-8eab-4013-b636-0162d82363c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm"
Sep 18 16:06:29.475990 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.475962    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5417439-8eab-4013-b636-0162d82363c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm\" (UID: \"d5417439-8eab-4013-b636-0162d82363c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm"
Sep 18 16:06:29.476598 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.476579    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5417439-8eab-4013-b636-0162d82363c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm\" (UID: \"d5417439-8eab-4013-b636-0162d82363c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm"
Sep 18 16:06:29.617513 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.617426    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm"
Sep 18 16:06:29.886059 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:29.886030    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm"]
Sep 18 16:06:29.888556 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:06:29.888527    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5417439_8eab_4013_b636_0162d82363c0.slice/crio-cc9755602e571fa58eea48cc8c5b895667619215213bac15401ddc4ab30ed9bd WatchSource:0}: Error finding container cc9755602e571fa58eea48cc8c5b895667619215213bac15401ddc4ab30ed9bd: Status 404 returned error can't find the container with id cc9755602e571fa58eea48cc8c5b895667619215213bac15401ddc4ab30ed9bd
Sep 18 16:06:30.405911 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:30.405873    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm" event={"ID":"d5417439-8eab-4013-b636-0162d82363c0","Type":"ContainerStarted","Data":"b5261e67c064eb098434dea8b491d4024c1c9c7980fa469d84ad4328c1ea097d"}
Sep 18 16:06:30.406302 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:30.405941    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm" event={"ID":"d5417439-8eab-4013-b636-0162d82363c0","Type":"ContainerStarted","Data":"cc9755602e571fa58eea48cc8c5b895667619215213bac15401ddc4ab30ed9bd"}
Sep 18 16:06:30.423454 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:30.423407    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm" podStartSLOduration=1.423394706 podStartE2EDuration="1.423394706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:06:30.421968664 +0000 UTC m=+1933.967259339" watchObservedRunningTime="2026-09-18 16:06:30.423394706 +0000 UTC m=+1933.968685380"
Sep 18 16:06:31.411811 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:31.411780    2561 generic.go:356] "Generic (PLEG): container finished" podID="d5417439-8eab-4013-b636-0162d82363c0" containerID="b5261e67c064eb098434dea8b491d4024c1c9c7980fa469d84ad4328c1ea097d" exitCode=0
Sep 18 16:06:31.412181 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:31.411818    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm" event={"ID":"d5417439-8eab-4013-b636-0162d82363c0","Type":"ContainerDied","Data":"b5261e67c064eb098434dea8b491d4024c1c9c7980fa469d84ad4328c1ea097d"}
Sep 18 16:06:32.545918 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:32.545892    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm"
Sep 18 16:06:32.605441 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:32.605413    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5417439-8eab-4013-b636-0162d82363c0-maas-api-service-ca\") pod \"d5417439-8eab-4013-b636-0162d82363c0\" (UID: \"d5417439-8eab-4013-b636-0162d82363c0\") "
Sep 18 16:06:32.605762 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:32.605740    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5417439-8eab-4013-b636-0162d82363c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5417439-8eab-4013-b636-0162d82363c0" (UID: "d5417439-8eab-4013-b636-0162d82363c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:06:32.706291 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:32.706270    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5417439-8eab-4013-b636-0162d82363c0-maas-api-service-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:06:33.422174 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:33.422147    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm"
Sep 18 16:06:33.422356 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:33.422140    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm" event={"ID":"d5417439-8eab-4013-b636-0162d82363c0","Type":"ContainerDied","Data":"cc9755602e571fa58eea48cc8c5b895667619215213bac15401ddc4ab30ed9bd"}
Sep 18 16:06:33.422356 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:33.422265    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc9755602e571fa58eea48cc8c5b895667619215213bac15401ddc4ab30ed9bd"
Sep 18 16:06:40.101416 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:40.101371    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm"]
Sep 18 16:06:40.103863 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:40.103838    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4e81aa-pmdjm"]
Sep 18 16:06:40.379961 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:40.379882    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5417439-8eab-4013-b636-0162d82363c0" path="/var/lib/kubelet/pods/d5417439-8eab-4013-b636-0162d82363c0/volumes"
Sep 18 16:06:48.859229 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:48.859199    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-7fxd2"]
Sep 18 16:06:48.859707 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:48.859438    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2" podUID="478ebc8d-edd3-4c32-8ca6-35a48d4707f8" containerName="manager" containerID="cri-o://16ede73e6f5d06ef9a96e0ef0a13e81dea57dad010bb2a2c93d888cfc041e85a" gracePeriod=30
Sep 18 16:06:49.106195 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.106162    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 16:06:49.157578 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.157501    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-webhook-cert\") pod \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\" (UID: \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\") "
Sep 18 16:06:49.157578 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.157556    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzn62\" (UniqueName: \"kubernetes.io/projected/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-kube-api-access-gzn62\") pod \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\" (UID: \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\") "
Sep 18 16:06:49.157835 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.157614    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-metrics-certs\") pod \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\" (UID: \"478ebc8d-edd3-4c32-8ca6-35a48d4707f8\") "
Sep 18 16:06:49.159928 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.159897    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "478ebc8d-edd3-4c32-8ca6-35a48d4707f8" (UID: "478ebc8d-edd3-4c32-8ca6-35a48d4707f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:06:49.159928 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.159916    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "478ebc8d-edd3-4c32-8ca6-35a48d4707f8" (UID: "478ebc8d-edd3-4c32-8ca6-35a48d4707f8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:06:49.160081 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.160065    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-kube-api-access-gzn62" (OuterVolumeSpecName: "kube-api-access-gzn62") pod "478ebc8d-edd3-4c32-8ca6-35a48d4707f8" (UID: "478ebc8d-edd3-4c32-8ca6-35a48d4707f8"). InnerVolumeSpecName "kube-api-access-gzn62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:06:49.259080 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.259047    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-webhook-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:06:49.259080 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.259075    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzn62\" (UniqueName: \"kubernetes.io/projected/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-kube-api-access-gzn62\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:06:49.259080 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.259084    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/478ebc8d-edd3-4c32-8ca6-35a48d4707f8-metrics-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:06:49.513000 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.512962    2561 generic.go:356] "Generic (PLEG): container finished" podID="478ebc8d-edd3-4c32-8ca6-35a48d4707f8" containerID="16ede73e6f5d06ef9a96e0ef0a13e81dea57dad010bb2a2c93d888cfc041e85a" exitCode=0
Sep 18 16:06:49.513166 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.513008    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2"
Sep 18 16:06:49.513166 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.513010    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2" event={"ID":"478ebc8d-edd3-4c32-8ca6-35a48d4707f8","Type":"ContainerDied","Data":"16ede73e6f5d06ef9a96e0ef0a13e81dea57dad010bb2a2c93d888cfc041e85a"}
Sep 18 16:06:49.513166 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.513059    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-7fxd2" event={"ID":"478ebc8d-edd3-4c32-8ca6-35a48d4707f8","Type":"ContainerDied","Data":"f0a23490106b412faf59cf93db9e783136d238758ed4c619c0bec6ba06fd0087"}
Sep 18 16:06:49.513166 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.513089    2561 scope.go:117] "RemoveContainer" containerID="16ede73e6f5d06ef9a96e0ef0a13e81dea57dad010bb2a2c93d888cfc041e85a"
Sep 18 16:06:49.523613 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.523595    2561 scope.go:117] "RemoveContainer" containerID="16ede73e6f5d06ef9a96e0ef0a13e81dea57dad010bb2a2c93d888cfc041e85a"
Sep 18 16:06:49.523947 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:06:49.523928    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ede73e6f5d06ef9a96e0ef0a13e81dea57dad010bb2a2c93d888cfc041e85a\": container with ID starting with 16ede73e6f5d06ef9a96e0ef0a13e81dea57dad010bb2a2c93d888cfc041e85a not found: ID does not exist" containerID="16ede73e6f5d06ef9a96e0ef0a13e81dea57dad010bb2a2c93d888cfc041e85a"
Sep 18 16:06:49.524016 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.523956    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ede73e6f5d06ef9a96e0ef0a13e81dea57dad010bb2a2c93d888cfc041e85a"} err="failed to get container status \"16ede73e6f5d06ef9a96e0ef0a13e81dea57dad010bb2a2c93d888cfc041e85a\": rpc error: code = NotFound desc = could not find container \"16ede73e6f5d06ef9a96e0ef0a13e81dea57dad010bb2a2c93d888cfc041e85a\": container with ID starting with 16ede73e6f5d06ef9a96e0ef0a13e81dea57dad010bb2a2c93d888cfc041e85a not found: ID does not exist"
Sep 18 16:06:49.534295 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.534265    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-7fxd2"]
Sep 18 16:06:49.541037 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:49.541014    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-7fxd2"]
Sep 18 16:06:50.379604 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:06:50.379570    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="478ebc8d-edd3-4c32-8ca6-35a48d4707f8" path="/var/lib/kubelet/pods/478ebc8d-edd3-4c32-8ca6-35a48d4707f8/volumes"
Sep 18 16:07:18.669445 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:18.669379    2561 scope.go:117] "RemoveContainer" containerID="1a42384c85a56811c1eb3abdeba994ea2dbe673158b5e9f4d9c9e8682e00e514"
Sep 18 16:07:50.349231 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.349193    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-jzm4n"]
Sep 18 16:07:50.349753 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.349607    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5417439-8eab-4013-b636-0162d82363c0" containerName="revoke-keys"
Sep 18 16:07:50.349753 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.349618    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5417439-8eab-4013-b636-0162d82363c0" containerName="revoke-keys"
Sep 18 16:07:50.349753 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.349627    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="478ebc8d-edd3-4c32-8ca6-35a48d4707f8" containerName="manager"
Sep 18 16:07:50.349753 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.349633    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="478ebc8d-edd3-4c32-8ca6-35a48d4707f8" containerName="manager"
Sep 18 16:07:50.349753 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.349718    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5417439-8eab-4013-b636-0162d82363c0" containerName="revoke-keys"
Sep 18 16:07:50.349753 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.349730    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="478ebc8d-edd3-4c32-8ca6-35a48d4707f8" containerName="manager"
Sep 18 16:07:50.352713 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.352698    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:07:50.356759 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.356730    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 16:07:50.356759 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.356738    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qjgkv\""
Sep 18 16:07:50.356946 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.356739    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 16:07:50.356946 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.356865    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 16:07:50.367017 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.366996    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-jzm4n"]
Sep 18 16:07:50.409313 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.409285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2072eacc-4ef9-49ad-b275-8854ceafecfc-webhook-cert\") pod \"maas-controller-7f9bc5dc6f-jzm4n\" (UID: \"2072eacc-4ef9-49ad-b275-8854ceafecfc\") " pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:07:50.409453 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.409353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2072eacc-4ef9-49ad-b275-8854ceafecfc-metrics-certs\") pod \"maas-controller-7f9bc5dc6f-jzm4n\" (UID: \"2072eacc-4ef9-49ad-b275-8854ceafecfc\") " pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:07:50.409453 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.409412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgbqp\" (UniqueName: \"kubernetes.io/projected/2072eacc-4ef9-49ad-b275-8854ceafecfc-kube-api-access-xgbqp\") pod \"maas-controller-7f9bc5dc6f-jzm4n\" (UID: \"2072eacc-4ef9-49ad-b275-8854ceafecfc\") " pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:07:50.510798 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.510775    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2072eacc-4ef9-49ad-b275-8854ceafecfc-webhook-cert\") pod \"maas-controller-7f9bc5dc6f-jzm4n\" (UID: \"2072eacc-4ef9-49ad-b275-8854ceafecfc\") " pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:07:50.510909 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.510810    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2072eacc-4ef9-49ad-b275-8854ceafecfc-metrics-certs\") pod \"maas-controller-7f9bc5dc6f-jzm4n\" (UID: \"2072eacc-4ef9-49ad-b275-8854ceafecfc\") " pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:07:50.510909 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.510830    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgbqp\" (UniqueName: \"kubernetes.io/projected/2072eacc-4ef9-49ad-b275-8854ceafecfc-kube-api-access-xgbqp\") pod \"maas-controller-7f9bc5dc6f-jzm4n\" (UID: \"2072eacc-4ef9-49ad-b275-8854ceafecfc\") " pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:07:50.513225 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.513206    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2072eacc-4ef9-49ad-b275-8854ceafecfc-metrics-certs\") pod \"maas-controller-7f9bc5dc6f-jzm4n\" (UID: \"2072eacc-4ef9-49ad-b275-8854ceafecfc\") " pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:07:50.513322 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.513254    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2072eacc-4ef9-49ad-b275-8854ceafecfc-webhook-cert\") pod \"maas-controller-7f9bc5dc6f-jzm4n\" (UID: \"2072eacc-4ef9-49ad-b275-8854ceafecfc\") " pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:07:50.519875 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.519855    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgbqp\" (UniqueName: \"kubernetes.io/projected/2072eacc-4ef9-49ad-b275-8854ceafecfc-kube-api-access-xgbqp\") pod \"maas-controller-7f9bc5dc6f-jzm4n\" (UID: \"2072eacc-4ef9-49ad-b275-8854ceafecfc\") " pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:07:50.663362 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.663305    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:07:50.940859 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.940833    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-jzm4n"]
Sep 18 16:07:50.942756 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:07:50.942730    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2072eacc_4ef9_49ad_b275_8854ceafecfc.slice/crio-0535797e72ac5ca9225efadae8136c77b0998844312a313bdd803251248d65de WatchSource:0}: Error finding container 0535797e72ac5ca9225efadae8136c77b0998844312a313bdd803251248d65de: Status 404 returned error can't find the container with id 0535797e72ac5ca9225efadae8136c77b0998844312a313bdd803251248d65de
Sep 18 16:07:50.944094 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:50.944077    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:07:51.811757 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:51.811715    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n" event={"ID":"2072eacc-4ef9-49ad-b275-8854ceafecfc","Type":"ContainerStarted","Data":"59ac3a7c13f91e1f20d1d084d151a932ec4ef7dc3bfed7dfc7cf227d9f2a0733"}
Sep 18 16:07:51.812120 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:51.811780    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n" event={"ID":"2072eacc-4ef9-49ad-b275-8854ceafecfc","Type":"ContainerStarted","Data":"0535797e72ac5ca9225efadae8136c77b0998844312a313bdd803251248d65de"}
Sep 18 16:07:51.812120 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:51.811891    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:07:51.831293 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:07:51.831247    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n" podStartSLOduration=1.490685132 podStartE2EDuration="1.831233664s" totalImagesPullingTime="340.548532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:07:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:07:51.829657605 +0000 UTC m=+2015.374948280" watchObservedRunningTime="2026-09-18 16:07:51.831233664 +0000 UTC m=+2015.376524384"
Sep 18 16:08:02.818345 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:02.818308    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:08:10.073312 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.073265    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-jzm4n"]
Sep 18 16:08:10.073811 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.073553    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n" podUID="2072eacc-4ef9-49ad-b275-8854ceafecfc" containerName="manager" containerID="cri-o://59ac3a7c13f91e1f20d1d084d151a932ec4ef7dc3bfed7dfc7cf227d9f2a0733" gracePeriod=30
Sep 18 16:08:10.421913 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.421892    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:08:10.595531 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.595452    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgbqp\" (UniqueName: \"kubernetes.io/projected/2072eacc-4ef9-49ad-b275-8854ceafecfc-kube-api-access-xgbqp\") pod \"2072eacc-4ef9-49ad-b275-8854ceafecfc\" (UID: \"2072eacc-4ef9-49ad-b275-8854ceafecfc\") "
Sep 18 16:08:10.595531 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.595499    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2072eacc-4ef9-49ad-b275-8854ceafecfc-webhook-cert\") pod \"2072eacc-4ef9-49ad-b275-8854ceafecfc\" (UID: \"2072eacc-4ef9-49ad-b275-8854ceafecfc\") "
Sep 18 16:08:10.595531 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.595520    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2072eacc-4ef9-49ad-b275-8854ceafecfc-metrics-certs\") pod \"2072eacc-4ef9-49ad-b275-8854ceafecfc\" (UID: \"2072eacc-4ef9-49ad-b275-8854ceafecfc\") "
Sep 18 16:08:10.597656 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.597632    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2072eacc-4ef9-49ad-b275-8854ceafecfc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2072eacc-4ef9-49ad-b275-8854ceafecfc" (UID: "2072eacc-4ef9-49ad-b275-8854ceafecfc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:08:10.597794 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.597632    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2072eacc-4ef9-49ad-b275-8854ceafecfc-kube-api-access-xgbqp" (OuterVolumeSpecName: "kube-api-access-xgbqp") pod "2072eacc-4ef9-49ad-b275-8854ceafecfc" (UID: "2072eacc-4ef9-49ad-b275-8854ceafecfc"). InnerVolumeSpecName "kube-api-access-xgbqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:08:10.597794 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.597718    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2072eacc-4ef9-49ad-b275-8854ceafecfc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2072eacc-4ef9-49ad-b275-8854ceafecfc" (UID: "2072eacc-4ef9-49ad-b275-8854ceafecfc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:08:10.696648 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.696616    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgbqp\" (UniqueName: \"kubernetes.io/projected/2072eacc-4ef9-49ad-b275-8854ceafecfc-kube-api-access-xgbqp\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:08:10.696648 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.696642    2561 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2072eacc-4ef9-49ad-b275-8854ceafecfc-webhook-cert\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:08:10.696648 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.696651    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2072eacc-4ef9-49ad-b275-8854ceafecfc-metrics-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:08:10.903987 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.903894    2561 generic.go:356] "Generic (PLEG): container finished" podID="2072eacc-4ef9-49ad-b275-8854ceafecfc" containerID="59ac3a7c13f91e1f20d1d084d151a932ec4ef7dc3bfed7dfc7cf227d9f2a0733" exitCode=0
Sep 18 16:08:10.903987 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.903935    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n"
Sep 18 16:08:10.903987 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.903974    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n" event={"ID":"2072eacc-4ef9-49ad-b275-8854ceafecfc","Type":"ContainerDied","Data":"59ac3a7c13f91e1f20d1d084d151a932ec4ef7dc3bfed7dfc7cf227d9f2a0733"}
Sep 18 16:08:10.904272 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.904033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-jzm4n" event={"ID":"2072eacc-4ef9-49ad-b275-8854ceafecfc","Type":"ContainerDied","Data":"0535797e72ac5ca9225efadae8136c77b0998844312a313bdd803251248d65de"}
Sep 18 16:08:10.904272 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.904063    2561 scope.go:117] "RemoveContainer" containerID="59ac3a7c13f91e1f20d1d084d151a932ec4ef7dc3bfed7dfc7cf227d9f2a0733"
Sep 18 16:08:10.914116 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.914100    2561 scope.go:117] "RemoveContainer" containerID="59ac3a7c13f91e1f20d1d084d151a932ec4ef7dc3bfed7dfc7cf227d9f2a0733"
Sep 18 16:08:10.914390 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:08:10.914368    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ac3a7c13f91e1f20d1d084d151a932ec4ef7dc3bfed7dfc7cf227d9f2a0733\": container with ID starting with 59ac3a7c13f91e1f20d1d084d151a932ec4ef7dc3bfed7dfc7cf227d9f2a0733 not found: ID does not exist" containerID="59ac3a7c13f91e1f20d1d084d151a932ec4ef7dc3bfed7dfc7cf227d9f2a0733"
Sep 18 16:08:10.914461 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.914397    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ac3a7c13f91e1f20d1d084d151a932ec4ef7dc3bfed7dfc7cf227d9f2a0733"} err="failed to get container status \"59ac3a7c13f91e1f20d1d084d151a932ec4ef7dc3bfed7dfc7cf227d9f2a0733\": rpc error: code = NotFound desc = could not find container \"59ac3a7c13f91e1f20d1d084d151a932ec4ef7dc3bfed7dfc7cf227d9f2a0733\": container with ID starting with 59ac3a7c13f91e1f20d1d084d151a932ec4ef7dc3bfed7dfc7cf227d9f2a0733 not found: ID does not exist"
Sep 18 16:08:10.926899 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.926879    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-jzm4n"]
Sep 18 16:08:10.930804 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:10.930785    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-jzm4n"]
Sep 18 16:08:11.771326 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:11.771293    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-tmll4"]
Sep 18 16:08:11.771730 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:11.771705    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2072eacc-4ef9-49ad-b275-8854ceafecfc" containerName="manager"
Sep 18 16:08:11.771730 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:11.771719    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2072eacc-4ef9-49ad-b275-8854ceafecfc" containerName="manager"
Sep 18 16:08:11.771814 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:11.771792    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2072eacc-4ef9-49ad-b275-8854ceafecfc" containerName="manager"
Sep 18 16:08:11.776124 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:11.776105    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4"
Sep 18 16:08:11.786091 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:11.786074    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 18 16:08:11.786927 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:11.786910    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-qjgkv\""
Sep 18 16:08:11.787006 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:11.786950    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 18 16:08:11.787043 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:11.786950    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 18 16:08:11.802760 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:11.802740    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-tmll4"]
Sep 18 16:08:11.908789 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:11.908762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f64c4e-417b-420b-99e9-02dcd73cdeaa-metrics-certs\") pod \"maas-controller-7f9bc5dc6f-tmll4\" (UID: \"e5f64c4e-417b-420b-99e9-02dcd73cdeaa\") " pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4"
Sep 18 16:08:11.908973 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:11.908860    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5f64c4e-417b-420b-99e9-02dcd73cdeaa-webhook-cert\") pod \"maas-controller-7f9bc5dc6f-tmll4\" (UID: \"e5f64c4e-417b-420b-99e9-02dcd73cdeaa\") " pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4"
Sep 18 16:08:11.908973 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:11.908900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvzqg\" (UniqueName: \"kubernetes.io/projected/e5f64c4e-417b-420b-99e9-02dcd73cdeaa-kube-api-access-kvzqg\") pod \"maas-controller-7f9bc5dc6f-tmll4\" (UID: \"e5f64c4e-417b-420b-99e9-02dcd73cdeaa\") " pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4"
Sep 18 16:08:12.010061 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:12.010033    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5f64c4e-417b-420b-99e9-02dcd73cdeaa-webhook-cert\") pod \"maas-controller-7f9bc5dc6f-tmll4\" (UID: \"e5f64c4e-417b-420b-99e9-02dcd73cdeaa\") " pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4"
Sep 18 16:08:12.010187 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:12.010065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvzqg\" (UniqueName: \"kubernetes.io/projected/e5f64c4e-417b-420b-99e9-02dcd73cdeaa-kube-api-access-kvzqg\") pod \"maas-controller-7f9bc5dc6f-tmll4\" (UID: \"e5f64c4e-417b-420b-99e9-02dcd73cdeaa\") " pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4"
Sep 18 16:08:12.010187 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:12.010120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f64c4e-417b-420b-99e9-02dcd73cdeaa-metrics-certs\") pod \"maas-controller-7f9bc5dc6f-tmll4\" (UID: \"e5f64c4e-417b-420b-99e9-02dcd73cdeaa\") " pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4"
Sep 18 16:08:12.012552 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:12.012524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5f64c4e-417b-420b-99e9-02dcd73cdeaa-metrics-certs\") pod \"maas-controller-7f9bc5dc6f-tmll4\" (UID: \"e5f64c4e-417b-420b-99e9-02dcd73cdeaa\") " pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4"
Sep 18 16:08:12.012677 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:12.012562    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e5f64c4e-417b-420b-99e9-02dcd73cdeaa-webhook-cert\") pod \"maas-controller-7f9bc5dc6f-tmll4\" (UID: \"e5f64c4e-417b-420b-99e9-02dcd73cdeaa\") " pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4"
Sep 18 16:08:12.018537 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:12.018511    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvzqg\" (UniqueName: \"kubernetes.io/projected/e5f64c4e-417b-420b-99e9-02dcd73cdeaa-kube-api-access-kvzqg\") pod \"maas-controller-7f9bc5dc6f-tmll4\" (UID: \"e5f64c4e-417b-420b-99e9-02dcd73cdeaa\") " pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4"
Sep 18 16:08:12.085518 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:12.085457    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4"
Sep 18 16:08:12.360832 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:12.360805    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f9bc5dc6f-tmll4"]
Sep 18 16:08:12.363055 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:08:12.363027    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f64c4e_417b_420b_99e9_02dcd73cdeaa.slice/crio-f4bcf396f05de908865049209afe4181e9efc5425022e0e107e99449c12db019 WatchSource:0}: Error finding container f4bcf396f05de908865049209afe4181e9efc5425022e0e107e99449c12db019: Status 404 returned error can't find the container with id f4bcf396f05de908865049209afe4181e9efc5425022e0e107e99449c12db019
Sep 18 16:08:12.379910 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:12.379884    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2072eacc-4ef9-49ad-b275-8854ceafecfc" path="/var/lib/kubelet/pods/2072eacc-4ef9-49ad-b275-8854ceafecfc/volumes"
Sep 18 16:08:12.916186 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:12.916157    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4" event={"ID":"e5f64c4e-417b-420b-99e9-02dcd73cdeaa","Type":"ContainerStarted","Data":"e1e5672f675b8d73e8e7c51f8b1841705bcd9ae32aedebda86edfa87c6227edb"}
Sep 18 16:08:12.916583 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:12.916205    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4" event={"ID":"e5f64c4e-417b-420b-99e9-02dcd73cdeaa","Type":"ContainerStarted","Data":"f4bcf396f05de908865049209afe4181e9efc5425022e0e107e99449c12db019"}
Sep 18 16:08:12.916583 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:12.916272    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4"
Sep 18 16:08:12.954615 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:12.954574    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4" podStartSLOduration=1.610071166 podStartE2EDuration="1.954560896s" totalImagesPullingTime="344.48973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:08:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:08:12.953633483 +0000 UTC m=+2036.498924158" watchObservedRunningTime="2026-09-18 16:08:12.954560896 +0000 UTC m=+2036.499851571"
Sep 18 16:08:23.923140 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:08:23.923103    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f9bc5dc6f-tmll4"
Sep 18 16:09:18.496327 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:09:18.496299    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 16:09:18.504593 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:09:18.504573    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 16:11:02.243730 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.243645    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"]
Sep 18 16:11:02.247487 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.247465    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.250310 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.250280    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-a0240115-kserve-self-signed-certs\""
Sep 18 16:11:02.260828 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.260800    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"]
Sep 18 16:11:02.356130 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.356087    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-model-cache\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.356130 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.356129    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734a1775-e140-477a-9a91-98127e7d3b2f-tls-certs\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.356396 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.356203    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-dshm\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.356396 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.356321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-tmp-dir\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.356396 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.356342    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-home\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.356396 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.356367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8tj5\" (UniqueName: \"kubernetes.io/projected/734a1775-e140-477a-9a91-98127e7d3b2f-kube-api-access-p8tj5\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.457304 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.457267    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-tmp-dir\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.457304 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.457313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-home\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.457587 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.457333    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8tj5\" (UniqueName: \"kubernetes.io/projected/734a1775-e140-477a-9a91-98127e7d3b2f-kube-api-access-p8tj5\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.457587 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.457358    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-model-cache\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.457587 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.457375    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734a1775-e140-477a-9a91-98127e7d3b2f-tls-certs\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.457587 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.457425    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-dshm\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.457886 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.457775    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-tmp-dir\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.457886 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.457834    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-model-cache\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.457997 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.457964    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-home\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.459812 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.459795    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-dshm\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.460184 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.460167    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734a1775-e140-477a-9a91-98127e7d3b2f-tls-certs\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.466841 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.466817    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8tj5\" (UniqueName: \"kubernetes.io/projected/734a1775-e140-477a-9a91-98127e7d3b2f-kube-api-access-p8tj5\") pod \"e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") " pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.556810 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.556713    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:02.830411 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:02.830386    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"]
Sep 18 16:11:02.832207 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:11:02.832175    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod734a1775_e140_477a_9a91_98127e7d3b2f.slice/crio-ee228e329a1520994c69bb12f33e6b839ff52a58450cc6f1aa96bc183f3268ca WatchSource:0}: Error finding container ee228e329a1520994c69bb12f33e6b839ff52a58450cc6f1aa96bc183f3268ca: Status 404 returned error can't find the container with id ee228e329a1520994c69bb12f33e6b839ff52a58450cc6f1aa96bc183f3268ca
Sep 18 16:11:03.746442 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:03.746406    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2" event={"ID":"734a1775-e140-477a-9a91-98127e7d3b2f","Type":"ContainerStarted","Data":"0344dd67faefbe2f75550b767eb34f139e68b8a16eb6e1f70d419661fb357899"}
Sep 18 16:11:03.746442 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:03.746452    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2" event={"ID":"734a1775-e140-477a-9a91-98127e7d3b2f","Type":"ContainerStarted","Data":"ee228e329a1520994c69bb12f33e6b839ff52a58450cc6f1aa96bc183f3268ca"}
Sep 18 16:11:03.780133 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:03.780078    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2" podStartSLOduration=1.780060322 podStartE2EDuration="1.780060322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:11:03.778465335 +0000 UTC m=+2207.323756009" watchObservedRunningTime="2026-09-18 16:11:03.780060322 +0000 UTC m=+2207.325350995"
Sep 18 16:11:12.557127 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:12.557076    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:12.557127 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:12.557140    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:12.570186 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:12.570157    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:12.802783 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:12.802752    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:11:55.820942 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:11:55.820906    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"]
Sep 18 16:12:10.840842 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:10.840796    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2" podUID="734a1775-e140-477a-9a91-98127e7d3b2f" containerName="main" containerID="cri-o://0344dd67faefbe2f75550b767eb34f139e68b8a16eb6e1f70d419661fb357899" gracePeriod=45
Sep 18 16:12:11.079788 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.079763    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:12:11.085717 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.085685    2561 generic.go:356] "Generic (PLEG): container finished" podID="734a1775-e140-477a-9a91-98127e7d3b2f" containerID="0344dd67faefbe2f75550b767eb34f139e68b8a16eb6e1f70d419661fb357899" exitCode=0
Sep 18 16:12:11.085872 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.085736    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"
Sep 18 16:12:11.085872 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.085776    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2" event={"ID":"734a1775-e140-477a-9a91-98127e7d3b2f","Type":"ContainerDied","Data":"0344dd67faefbe2f75550b767eb34f139e68b8a16eb6e1f70d419661fb357899"}
Sep 18 16:12:11.085872 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.085816    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2" event={"ID":"734a1775-e140-477a-9a91-98127e7d3b2f","Type":"ContainerDied","Data":"ee228e329a1520994c69bb12f33e6b839ff52a58450cc6f1aa96bc183f3268ca"}
Sep 18 16:12:11.085872 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.085834    2561 scope.go:117] "RemoveContainer" containerID="0344dd67faefbe2f75550b767eb34f139e68b8a16eb6e1f70d419661fb357899"
Sep 18 16:12:11.094484 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.094460    2561 scope.go:117] "RemoveContainer" containerID="0344dd67faefbe2f75550b767eb34f139e68b8a16eb6e1f70d419661fb357899"
Sep 18 16:12:11.094804 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:12:11.094784    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0344dd67faefbe2f75550b767eb34f139e68b8a16eb6e1f70d419661fb357899\": container with ID starting with 0344dd67faefbe2f75550b767eb34f139e68b8a16eb6e1f70d419661fb357899 not found: ID does not exist" containerID="0344dd67faefbe2f75550b767eb34f139e68b8a16eb6e1f70d419661fb357899"
Sep 18 16:12:11.094881 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.094812    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0344dd67faefbe2f75550b767eb34f139e68b8a16eb6e1f70d419661fb357899"} err="failed to get container status \"0344dd67faefbe2f75550b767eb34f139e68b8a16eb6e1f70d419661fb357899\": rpc error: code = NotFound desc = could not find container \"0344dd67faefbe2f75550b767eb34f139e68b8a16eb6e1f70d419661fb357899\": container with ID starting with 0344dd67faefbe2f75550b767eb34f139e68b8a16eb6e1f70d419661fb357899 not found: ID does not exist"
Sep 18 16:12:11.211186 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.211152    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8tj5\" (UniqueName: \"kubernetes.io/projected/734a1775-e140-477a-9a91-98127e7d3b2f-kube-api-access-p8tj5\") pod \"734a1775-e140-477a-9a91-98127e7d3b2f\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") "
Sep 18 16:12:11.211391 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.211195    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-dshm\") pod \"734a1775-e140-477a-9a91-98127e7d3b2f\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") "
Sep 18 16:12:11.211391 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.211255    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-tmp-dir\") pod \"734a1775-e140-477a-9a91-98127e7d3b2f\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") "
Sep 18 16:12:11.211391 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.211295    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734a1775-e140-477a-9a91-98127e7d3b2f-tls-certs\") pod \"734a1775-e140-477a-9a91-98127e7d3b2f\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") "
Sep 18 16:12:11.211391 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.211311    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-home\") pod \"734a1775-e140-477a-9a91-98127e7d3b2f\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") "
Sep 18 16:12:11.211623 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.211492    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "734a1775-e140-477a-9a91-98127e7d3b2f" (UID: "734a1775-e140-477a-9a91-98127e7d3b2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:12:11.211623 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.211519    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-model-cache\") pod \"734a1775-e140-477a-9a91-98127e7d3b2f\" (UID: \"734a1775-e140-477a-9a91-98127e7d3b2f\") "
Sep 18 16:12:11.211623 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.211570    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-home" (OuterVolumeSpecName: "home") pod "734a1775-e140-477a-9a91-98127e7d3b2f" (UID: "734a1775-e140-477a-9a91-98127e7d3b2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:12:11.211829 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.211632    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-model-cache" (OuterVolumeSpecName: "model-cache") pod "734a1775-e140-477a-9a91-98127e7d3b2f" (UID: "734a1775-e140-477a-9a91-98127e7d3b2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:12:11.211972 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.211945    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-tmp-dir\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:12:11.212046 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.211974    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-home\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:12:11.212046 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.211983    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-model-cache\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:12:11.213606 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.213569    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734a1775-e140-477a-9a91-98127e7d3b2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "734a1775-e140-477a-9a91-98127e7d3b2f" (UID: "734a1775-e140-477a-9a91-98127e7d3b2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:12:11.213606 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.213578    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-dshm" (OuterVolumeSpecName: "dshm") pod "734a1775-e140-477a-9a91-98127e7d3b2f" (UID: "734a1775-e140-477a-9a91-98127e7d3b2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:12:11.213765 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.213682    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734a1775-e140-477a-9a91-98127e7d3b2f-kube-api-access-p8tj5" (OuterVolumeSpecName: "kube-api-access-p8tj5") pod "734a1775-e140-477a-9a91-98127e7d3b2f" (UID: "734a1775-e140-477a-9a91-98127e7d3b2f"). InnerVolumeSpecName "kube-api-access-p8tj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:12:11.313321 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.313283    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8tj5\" (UniqueName: \"kubernetes.io/projected/734a1775-e140-477a-9a91-98127e7d3b2f-kube-api-access-p8tj5\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:12:11.313321 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.313312    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/734a1775-e140-477a-9a91-98127e7d3b2f-dshm\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:12:11.313321 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.313321    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/734a1775-e140-477a-9a91-98127e7d3b2f-tls-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:12:11.422556 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.422521    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"]
Sep 18 16:12:11.435046 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:11.435016    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-a0240115-kserve-847bcfc959-4vbj2"]
Sep 18 16:12:12.380789 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:12.380753    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="734a1775-e140-477a-9a91-98127e7d3b2f" path="/var/lib/kubelet/pods/734a1775-e140-477a-9a91-98127e7d3b2f/volumes"
Sep 18 16:12:15.212955 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.212914    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"]
Sep 18 16:12:15.213341 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.213303    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="734a1775-e140-477a-9a91-98127e7d3b2f" containerName="main"
Sep 18 16:12:15.213341 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.213313    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="734a1775-e140-477a-9a91-98127e7d3b2f" containerName="main"
Sep 18 16:12:15.213422 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.213399    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="734a1775-e140-477a-9a91-98127e7d3b2f" containerName="main"
Sep 18 16:12:15.218104 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.218076    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.221874 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.221850    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hp5gf\""
Sep 18 16:12:15.222891 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.222874    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 16:12:15.222988 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.222948    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 16:12:15.240284 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.240250    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"]
Sep 18 16:12:15.349616 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.349578    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-metrics-certs\") pod \"maas-api-59f6566bbc-tdkn2\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.349851 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.349653    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-maas-api-tls\") pod \"maas-api-59f6566bbc-tdkn2\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.349851 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.349759    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s2bj\" (UniqueName: \"kubernetes.io/projected/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-kube-api-access-8s2bj\") pod \"maas-api-59f6566bbc-tdkn2\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.349851 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.349793    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-openshift-service-ca\") pod \"maas-api-59f6566bbc-tdkn2\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.451137 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.451099    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s2bj\" (UniqueName: \"kubernetes.io/projected/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-kube-api-access-8s2bj\") pod \"maas-api-59f6566bbc-tdkn2\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.451137 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.451139    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-openshift-service-ca\") pod \"maas-api-59f6566bbc-tdkn2\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.451354 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.451233    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-metrics-certs\") pod \"maas-api-59f6566bbc-tdkn2\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.451354 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.451308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-maas-api-tls\") pod \"maas-api-59f6566bbc-tdkn2\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.451915 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.451882    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-openshift-service-ca\") pod \"maas-api-59f6566bbc-tdkn2\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.453942 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.453917    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-maas-api-tls\") pod \"maas-api-59f6566bbc-tdkn2\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.454088 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.454059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-metrics-certs\") pod \"maas-api-59f6566bbc-tdkn2\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.464787 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.464717    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s2bj\" (UniqueName: \"kubernetes.io/projected/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-kube-api-access-8s2bj\") pod \"maas-api-59f6566bbc-tdkn2\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") " pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.529041 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.529006    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:15.809766 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:15.809737    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"]
Sep 18 16:12:15.812538 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:12:15.812508    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2d62e5_590b_41f4_909a_e51aa28a5e2e.slice/crio-6855444d6278324573352dca7a4cc04011b05f522e10fa1bd5b2333145371300 WatchSource:0}: Error finding container 6855444d6278324573352dca7a4cc04011b05f522e10fa1bd5b2333145371300: Status 404 returned error can't find the container with id 6855444d6278324573352dca7a4cc04011b05f522e10fa1bd5b2333145371300
Sep 18 16:12:16.115177 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:16.115094    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2" event={"ID":"7d2d62e5-590b-41f4-909a-e51aa28a5e2e","Type":"ContainerStarted","Data":"6855444d6278324573352dca7a4cc04011b05f522e10fa1bd5b2333145371300"}
Sep 18 16:12:17.122450 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:17.122411    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2" event={"ID":"7d2d62e5-590b-41f4-909a-e51aa28a5e2e","Type":"ContainerStarted","Data":"a68037e0a193d96b53a1bfe270f7bdb1c76fd693e94beff03ecb52610a449f6f"}
Sep 18 16:12:17.122910 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:17.122506    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:17.148075 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:17.148019    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2" podStartSLOduration=1.7540703610000001 podStartE2EDuration="2.148004219s" totalImagesPullingTime="393.933858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:12:17.146994075 +0000 UTC m=+2280.692284748" watchObservedRunningTime="2026-09-18 16:12:17.148004219 +0000 UTC m=+2280.693294918"
Sep 18 16:12:23.134085 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.134056    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:23.368359 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.368320    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"]
Sep 18 16:12:23.373081 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.373060    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.413032 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.412961    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"]
Sep 18 16:12:23.535141 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.535101    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9112f09-eb39-43ec-adcb-0f40d148ecc6-metrics-certs\") pod \"maas-api-86d9ddc875-dq8bl\" (UID: \"a9112f09-eb39-43ec-adcb-0f40d148ecc6\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.535328 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.535159    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wthb\" (UniqueName: \"kubernetes.io/projected/a9112f09-eb39-43ec-adcb-0f40d148ecc6-kube-api-access-7wthb\") pod \"maas-api-86d9ddc875-dq8bl\" (UID: \"a9112f09-eb39-43ec-adcb-0f40d148ecc6\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.535390 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.535331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9112f09-eb39-43ec-adcb-0f40d148ecc6-maas-api-tls\") pod \"maas-api-86d9ddc875-dq8bl\" (UID: \"a9112f09-eb39-43ec-adcb-0f40d148ecc6\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.535390 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.535363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9112f09-eb39-43ec-adcb-0f40d148ecc6-openshift-service-ca\") pod \"maas-api-86d9ddc875-dq8bl\" (UID: \"a9112f09-eb39-43ec-adcb-0f40d148ecc6\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.637029 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.636992    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wthb\" (UniqueName: \"kubernetes.io/projected/a9112f09-eb39-43ec-adcb-0f40d148ecc6-kube-api-access-7wthb\") pod \"maas-api-86d9ddc875-dq8bl\" (UID: \"a9112f09-eb39-43ec-adcb-0f40d148ecc6\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.637211 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.637065    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9112f09-eb39-43ec-adcb-0f40d148ecc6-maas-api-tls\") pod \"maas-api-86d9ddc875-dq8bl\" (UID: \"a9112f09-eb39-43ec-adcb-0f40d148ecc6\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.637211 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.637083    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9112f09-eb39-43ec-adcb-0f40d148ecc6-openshift-service-ca\") pod \"maas-api-86d9ddc875-dq8bl\" (UID: \"a9112f09-eb39-43ec-adcb-0f40d148ecc6\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.637211 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.637124    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9112f09-eb39-43ec-adcb-0f40d148ecc6-metrics-certs\") pod \"maas-api-86d9ddc875-dq8bl\" (UID: \"a9112f09-eb39-43ec-adcb-0f40d148ecc6\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.637817 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.637792    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9112f09-eb39-43ec-adcb-0f40d148ecc6-openshift-service-ca\") pod \"maas-api-86d9ddc875-dq8bl\" (UID: \"a9112f09-eb39-43ec-adcb-0f40d148ecc6\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.639634 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.639607    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9112f09-eb39-43ec-adcb-0f40d148ecc6-metrics-certs\") pod \"maas-api-86d9ddc875-dq8bl\" (UID: \"a9112f09-eb39-43ec-adcb-0f40d148ecc6\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.639746 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.639691    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9112f09-eb39-43ec-adcb-0f40d148ecc6-maas-api-tls\") pod \"maas-api-86d9ddc875-dq8bl\" (UID: \"a9112f09-eb39-43ec-adcb-0f40d148ecc6\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.648558 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.648538    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wthb\" (UniqueName: \"kubernetes.io/projected/a9112f09-eb39-43ec-adcb-0f40d148ecc6-kube-api-access-7wthb\") pod \"maas-api-86d9ddc875-dq8bl\" (UID: \"a9112f09-eb39-43ec-adcb-0f40d148ecc6\") " pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.683498 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.683470    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:23.975817 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:23.975791    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"]
Sep 18 16:12:23.976977 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:12:23.976947    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9112f09_eb39_43ec_adcb_0f40d148ecc6.slice/crio-404ebe0c6bf1f76e9f7fcbf4475ad03a5f3b9b992ba0db283f285ee4b2064d64 WatchSource:0}: Error finding container 404ebe0c6bf1f76e9f7fcbf4475ad03a5f3b9b992ba0db283f285ee4b2064d64: Status 404 returned error can't find the container with id 404ebe0c6bf1f76e9f7fcbf4475ad03a5f3b9b992ba0db283f285ee4b2064d64
Sep 18 16:12:24.159394 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:24.159358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl" event={"ID":"a9112f09-eb39-43ec-adcb-0f40d148ecc6","Type":"ContainerStarted","Data":"404ebe0c6bf1f76e9f7fcbf4475ad03a5f3b9b992ba0db283f285ee4b2064d64"}
Sep 18 16:12:25.166039 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:25.166000    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl" event={"ID":"a9112f09-eb39-43ec-adcb-0f40d148ecc6","Type":"ContainerStarted","Data":"1028464c130033a8aa760e58a1186b5721e891eb54916acb9d5e4125d1037d5f"}
Sep 18 16:12:25.166459 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:25.166071    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:25.188015 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:25.187960    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl" podStartSLOduration=1.916549239 podStartE2EDuration="2.187944279s" totalImagesPullingTime="271.39504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:12:25.185307056 +0000 UTC m=+2288.730597729" watchObservedRunningTime="2026-09-18 16:12:25.187944279 +0000 UTC m=+2288.733234953"
Sep 18 16:12:31.176675 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.176631    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86d9ddc875-dq8bl"
Sep 18 16:12:31.234452 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.234414    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"]
Sep 18 16:12:31.234735 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.234656    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2" podUID="7d2d62e5-590b-41f4-909a-e51aa28a5e2e" containerName="maas-api" containerID="cri-o://a68037e0a193d96b53a1bfe270f7bdb1c76fd693e94beff03ecb52610a449f6f" gracePeriod=30
Sep 18 16:12:31.483441 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.483414    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:31.511348 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.511313    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-maas-api-tls\") pod \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") "
Sep 18 16:12:31.511604 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.511419    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-openshift-service-ca\") pod \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") "
Sep 18 16:12:31.511604 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.511460    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s2bj\" (UniqueName: \"kubernetes.io/projected/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-kube-api-access-8s2bj\") pod \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") "
Sep 18 16:12:31.511604 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.511486    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-metrics-certs\") pod \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\" (UID: \"7d2d62e5-590b-41f4-909a-e51aa28a5e2e\") "
Sep 18 16:12:31.512131 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.512052    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7d2d62e5-590b-41f4-909a-e51aa28a5e2e" (UID: "7d2d62e5-590b-41f4-909a-e51aa28a5e2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:12:31.514628 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.514572    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7d2d62e5-590b-41f4-909a-e51aa28a5e2e" (UID: "7d2d62e5-590b-41f4-909a-e51aa28a5e2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:12:31.514628 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.514616    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-kube-api-access-8s2bj" (OuterVolumeSpecName: "kube-api-access-8s2bj") pod "7d2d62e5-590b-41f4-909a-e51aa28a5e2e" (UID: "7d2d62e5-590b-41f4-909a-e51aa28a5e2e"). InnerVolumeSpecName "kube-api-access-8s2bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:12:31.515116 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.515090    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7d2d62e5-590b-41f4-909a-e51aa28a5e2e" (UID: "7d2d62e5-590b-41f4-909a-e51aa28a5e2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:12:31.613029 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.612993    2561 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-maas-api-tls\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:12:31.613029 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.613021    2561 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-openshift-service-ca\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:12:31.613029 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.613030    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s2bj\" (UniqueName: \"kubernetes.io/projected/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-kube-api-access-8s2bj\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:12:31.613029 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:31.613039    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d2d62e5-590b-41f4-909a-e51aa28a5e2e-metrics-certs\") on node \"ip-10-0-133-137.ec2.internal\" DevicePath \"\""
Sep 18 16:12:32.202688 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:32.202630    2561 generic.go:356] "Generic (PLEG): container finished" podID="7d2d62e5-590b-41f4-909a-e51aa28a5e2e" containerID="a68037e0a193d96b53a1bfe270f7bdb1c76fd693e94beff03ecb52610a449f6f" exitCode=0
Sep 18 16:12:32.203173 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:32.202702    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"
Sep 18 16:12:32.203173 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:32.202715    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2" event={"ID":"7d2d62e5-590b-41f4-909a-e51aa28a5e2e","Type":"ContainerDied","Data":"a68037e0a193d96b53a1bfe270f7bdb1c76fd693e94beff03ecb52610a449f6f"}
Sep 18 16:12:32.203173 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:32.202772    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2" event={"ID":"7d2d62e5-590b-41f4-909a-e51aa28a5e2e","Type":"ContainerDied","Data":"6855444d6278324573352dca7a4cc04011b05f522e10fa1bd5b2333145371300"}
Sep 18 16:12:32.203173 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:32.202795    2561 scope.go:117] "RemoveContainer" containerID="a68037e0a193d96b53a1bfe270f7bdb1c76fd693e94beff03ecb52610a449f6f"
Sep 18 16:12:32.213030 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:32.213012    2561 scope.go:117] "RemoveContainer" containerID="a68037e0a193d96b53a1bfe270f7bdb1c76fd693e94beff03ecb52610a449f6f"
Sep 18 16:12:32.213318 ip-10-0-133-137 kubenswrapper[2561]: E0918 16:12:32.213296    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68037e0a193d96b53a1bfe270f7bdb1c76fd693e94beff03ecb52610a449f6f\": container with ID starting with a68037e0a193d96b53a1bfe270f7bdb1c76fd693e94beff03ecb52610a449f6f not found: ID does not exist" containerID="a68037e0a193d96b53a1bfe270f7bdb1c76fd693e94beff03ecb52610a449f6f"
Sep 18 16:12:32.213366 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:32.213328    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68037e0a193d96b53a1bfe270f7bdb1c76fd693e94beff03ecb52610a449f6f"} err="failed to get container status \"a68037e0a193d96b53a1bfe270f7bdb1c76fd693e94beff03ecb52610a449f6f\": rpc error: code = NotFound desc = could not find container \"a68037e0a193d96b53a1bfe270f7bdb1c76fd693e94beff03ecb52610a449f6f\": container with ID starting with a68037e0a193d96b53a1bfe270f7bdb1c76fd693e94beff03ecb52610a449f6f not found: ID does not exist"
Sep 18 16:12:32.226808 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:32.226783    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"]
Sep 18 16:12:32.231976 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:32.231949    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59f6566bbc-tdkn2"]
Sep 18 16:12:32.379759 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:32.379726    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d2d62e5-590b-41f4-909a-e51aa28a5e2e" path="/var/lib/kubelet/pods/7d2d62e5-590b-41f4-909a-e51aa28a5e2e/volumes"
Sep 18 16:12:52.832960 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:52.832932    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gjx2r_cedebd21-fe8d-4e66-a7b4-f3dcf76e9aea/global-pull-secret-syncer/0.log"
Sep 18 16:12:53.141238 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:53.141164    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q2gdh_9a0c2c97-48b2-4eb6-848d-b997c400d782/konnectivity-agent/0.log"
Sep 18 16:12:53.310448 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:53.310414    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-137.ec2.internal_2079adb0a291ab6cf85aca3fef44d776/haproxy/0.log"
Sep 18 16:12:56.323212 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:56.323125    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d76dfb587-qlk8x_b6d9fb62-3526-42e7-8c2c-34a8b3eac395/authorino/0.log"
Sep 18 16:12:56.441095 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:56.441065    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-5plkr_7785d30c-b3a7-4c18-b334-1e6fd1f9bcdc/kuadrant-console-plugin/0.log"
Sep 18 16:12:58.535681 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:58.535637    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9349358a-28e2-4369-a8cc-4b8215982bad/alertmanager/0.log"
Sep 18 16:12:58.571854 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:58.571827    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9349358a-28e2-4369-a8cc-4b8215982bad/config-reloader/0.log"
Sep 18 16:12:58.608464 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:58.608431    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9349358a-28e2-4369-a8cc-4b8215982bad/kube-rbac-proxy-web/0.log"
Sep 18 16:12:58.637974 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:58.637946    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9349358a-28e2-4369-a8cc-4b8215982bad/kube-rbac-proxy/0.log"
Sep 18 16:12:58.662835 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:58.662812    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9349358a-28e2-4369-a8cc-4b8215982bad/kube-rbac-proxy-metric/0.log"
Sep 18 16:12:58.687027 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:58.686998    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9349358a-28e2-4369-a8cc-4b8215982bad/prom-label-proxy/0.log"
Sep 18 16:12:58.722061 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:58.721957    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9349358a-28e2-4369-a8cc-4b8215982bad/init-config-reloader/0.log"
Sep 18 16:12:58.764313 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:58.764283    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-czd8s_52dadbd4-3e28-42d7-9d38-b4343c8c37ab/cluster-monitoring-operator/0.log"
Sep 18 16:12:58.938969 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:58.938939    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-948xd_469cc9fd-0676-4be8-8d6f-c5031f06e0c2/node-exporter/0.log"
Sep 18 16:12:58.966511 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:58.966480    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-948xd_469cc9fd-0676-4be8-8d6f-c5031f06e0c2/kube-rbac-proxy/0.log"
Sep 18 16:12:58.993134 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:58.993111    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-948xd_469cc9fd-0676-4be8-8d6f-c5031f06e0c2/init-textfile/0.log"
Sep 18 16:12:59.628136 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:59.628106    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56b848874f-pl85h_54c6c283-2719-42f1-bfb1-009ede0820d0/telemeter-client/0.log"
Sep 18 16:12:59.663055 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:59.663034    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56b848874f-pl85h_54c6c283-2719-42f1-bfb1-009ede0820d0/reload/0.log"
Sep 18 16:12:59.693025 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:59.693003    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56b848874f-pl85h_54c6c283-2719-42f1-bfb1-009ede0820d0/kube-rbac-proxy/0.log"
Sep 18 16:12:59.758791 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:59.758767    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55fdc8b964-jlv2l_3e2b419c-6e64-475b-b85e-e6e297d8ef57/thanos-query/0.log"
Sep 18 16:12:59.781146 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:59.781119    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55fdc8b964-jlv2l_3e2b419c-6e64-475b-b85e-e6e297d8ef57/kube-rbac-proxy-web/0.log"
Sep 18 16:12:59.808391 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:59.808338    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55fdc8b964-jlv2l_3e2b419c-6e64-475b-b85e-e6e297d8ef57/kube-rbac-proxy/0.log"
Sep 18 16:12:59.851618 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:59.851593    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55fdc8b964-jlv2l_3e2b419c-6e64-475b-b85e-e6e297d8ef57/prom-label-proxy/0.log"
Sep 18 16:12:59.878014 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:59.877989    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55fdc8b964-jlv2l_3e2b419c-6e64-475b-b85e-e6e297d8ef57/kube-rbac-proxy-rules/0.log"
Sep 18 16:12:59.906711 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:12:59.906608    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55fdc8b964-jlv2l_3e2b419c-6e64-475b-b85e-e6e297d8ef57/kube-rbac-proxy-metrics/0.log"
Sep 18 16:13:02.116021 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:02.115988    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/1.log"
Sep 18 16:13:02.121372 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:02.121351    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-db9cx_577230bb-9bb3-42c4-9e6f-dda006b1552f/console-operator/2.log"
Sep 18 16:13:02.730771 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:02.730748    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c6b6bf45b-7h252_66e3bffa-358c-4cb3-875d-c9ad76901558/console/0.log"
Sep 18 16:13:02.775415 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:02.775390    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-hznvw_60f6b3de-1cb4-47e1-9d46-1c1292690318/download-server/0.log"
Sep 18 16:13:03.176396 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.176358    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"]
Sep 18 16:13:03.176811 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.176801    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2d62e5-590b-41f4-909a-e51aa28a5e2e" containerName="maas-api"
Sep 18 16:13:03.176880 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.176821    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2d62e5-590b-41f4-909a-e51aa28a5e2e" containerName="maas-api"
Sep 18 16:13:03.176936 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.176913    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d2d62e5-590b-41f4-909a-e51aa28a5e2e" containerName="maas-api"
Sep 18 16:13:03.180582 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.180561    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.183764 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.183742    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2zs2w\"/\"default-dockercfg-k85dh\""
Sep 18 16:13:03.184777 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.184758    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2zs2w\"/\"kube-root-ca.crt\""
Sep 18 16:13:03.184864 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.184782    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2zs2w\"/\"openshift-service-ca.crt\""
Sep 18 16:13:03.190016 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.189986    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"]
Sep 18 16:13:03.321238 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.321198    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06061725-c518-47cb-b6a8-0fc7cb32a02f-proc\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.321458 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.321244    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06061725-c518-47cb-b6a8-0fc7cb32a02f-lib-modules\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.321458 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.321346    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06061725-c518-47cb-b6a8-0fc7cb32a02f-sys\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.321458 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.321391    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06061725-c518-47cb-b6a8-0fc7cb32a02f-podres\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.321458 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.321425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtqtb\" (UniqueName: \"kubernetes.io/projected/06061725-c518-47cb-b6a8-0fc7cb32a02f-kube-api-access-wtqtb\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.422345 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.422308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06061725-c518-47cb-b6a8-0fc7cb32a02f-sys\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.422560 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.422355    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06061725-c518-47cb-b6a8-0fc7cb32a02f-podres\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.422560 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.422383    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtqtb\" (UniqueName: \"kubernetes.io/projected/06061725-c518-47cb-b6a8-0fc7cb32a02f-kube-api-access-wtqtb\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.422560 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.422442    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06061725-c518-47cb-b6a8-0fc7cb32a02f-proc\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.422560 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.422444    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06061725-c518-47cb-b6a8-0fc7cb32a02f-sys\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.422560 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.422464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06061725-c518-47cb-b6a8-0fc7cb32a02f-lib-modules\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.422560 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.422536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/06061725-c518-47cb-b6a8-0fc7cb32a02f-podres\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.422830 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.422566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/06061725-c518-47cb-b6a8-0fc7cb32a02f-proc\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.422830 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.422577    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06061725-c518-47cb-b6a8-0fc7cb32a02f-lib-modules\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.431189 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.431139    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtqtb\" (UniqueName: \"kubernetes.io/projected/06061725-c518-47cb-b6a8-0fc7cb32a02f-kube-api-access-wtqtb\") pod \"perf-node-gather-daemonset-7hwxn\" (UID: \"06061725-c518-47cb-b6a8-0fc7cb32a02f\") " pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.492259 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.492221    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:03.500901 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.500874    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-k4r5l_67e40f73-83c9-4f84-81d8-3b11ef6814d5/volume-data-source-validator/0.log"
Sep 18 16:13:03.770719 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:03.770691    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"]
Sep 18 16:13:03.772765 ip-10-0-133-137 kubenswrapper[2561]: W0918 16:13:03.772737    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod06061725_c518_47cb_b6a8_0fc7cb32a02f.slice/crio-e48ea765281db67d94f30422a4cc234e75f0bab524ab39406812bf65f34d574a WatchSource:0}: Error finding container e48ea765281db67d94f30422a4cc234e75f0bab524ab39406812bf65f34d574a: Status 404 returned error can't find the container with id e48ea765281db67d94f30422a4cc234e75f0bab524ab39406812bf65f34d574a
Sep 18 16:13:04.380224 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:04.380141    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn" event={"ID":"06061725-c518-47cb-b6a8-0fc7cb32a02f","Type":"ContainerStarted","Data":"44584b7c56329a2f3013ee1dd771c5134334d1a1b8794a3e4bf5fda590e3bc8b"}
Sep 18 16:13:04.380224 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:04.380200    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:04.380224 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:04.380216    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn" event={"ID":"06061725-c518-47cb-b6a8-0fc7cb32a02f","Type":"ContainerStarted","Data":"e48ea765281db67d94f30422a4cc234e75f0bab524ab39406812bf65f34d574a"}
Sep 18 16:13:04.395196 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:04.395145    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn" podStartSLOduration=1.395125817 podStartE2EDuration="1.395125817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:13:04.392970937 +0000 UTC m=+2327.938261612" watchObservedRunningTime="2026-09-18 16:13:04.395125817 +0000 UTC m=+2327.940416492"
Sep 18 16:13:04.950874 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:04.950832    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mmqzn_ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407/dns/0.log"
Sep 18 16:13:04.974580 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:04.974554    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mmqzn_ec9f2a0c-fd5b-4e1e-8c64-8131e8b89407/kube-rbac-proxy/0.log"
Sep 18 16:13:05.001400 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:05.001376    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-57tph_f2339bc3-3053-47db-b94c-dd9c926069a1/dns-node-resolver/0.log"
Sep 18 16:13:05.719738 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:05.719688    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m7hxp_4f4df3e2-04a4-4534-8c8d-aba5d753f87a/node-ca/0.log"
Sep 18 16:13:07.195383 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:07.195351    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fbb59494c-kv8cx_7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29/kube-auth-proxy/0.log"
Sep 18 16:13:07.315356 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:07.315331    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bf6bd4c64-ppwrg_3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436/router/0.log"
Sep 18 16:13:08.363430 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:08.363401    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rqkgz_84f7bd6b-71dd-48ea-b787-f0ed59ed4fa1/serve-healthcheck-canary/0.log"
Sep 18 16:13:08.862763 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:08.862734    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-2965g_42f4c511-6c8c-45f6-a86d-bb6f5d204022/insights-operator/0.log"
Sep 18 16:13:08.863452 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:08.863436    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-2965g_42f4c511-6c8c-45f6-a86d-bb6f5d204022/insights-operator/1.log"
Sep 18 16:13:09.044854 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:09.044826    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8rgc2_e4192f75-9684-4964-8021-2bb9c4d390c3/kube-rbac-proxy/0.log"
Sep 18 16:13:09.087919 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:09.087892    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8rgc2_e4192f75-9684-4964-8021-2bb9c4d390c3/exporter/0.log"
Sep 18 16:13:09.134897 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:09.134795    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8rgc2_e4192f75-9684-4964-8021-2bb9c4d390c3/extractor/0.log"
Sep 18 16:13:10.391655 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:10.391629    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2zs2w/perf-node-gather-daemonset-7hwxn"
Sep 18 16:13:11.912718 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:11.912684    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f9bc5dc6f-tmll4_e5f64c4e-417b-420b-99e9-02dcd73cdeaa/manager/0.log"
Sep 18 16:13:12.148856 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:12.148829    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54b8c8d665-mdph5_9e1bdfc0-5b16-44cd-8469-05647e693619/manager/0.log"
Sep 18 16:13:12.177610 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:12.177586    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54b8c8d665-mdph5_9e1bdfc0-5b16-44cd-8469-05647e693619/copy-manifests/0.log"
Sep 18 16:13:13.559537 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:13.559509    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fbb59494c-kv8cx_7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29/kube-auth-proxy/0.log"
Sep 18 16:13:13.630781 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:13.630700    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bf6bd4c64-ppwrg_3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436/router/0.log"
Sep 18 16:13:13.801924 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:13.801896    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-c59846bb8-dzjgk_15d48601-1baf-4949-98d5-d85083be374d/manager/0.log"
Sep 18 16:13:15.712486 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:15.712453    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7fbb59494c-kv8cx_7e5e0c89-88c9-4f9e-a4b3-6ea9c1f46f29/kube-auth-proxy/0.log"
Sep 18 16:13:15.791675 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:15.791645    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6bf6bd4c64-ppwrg_3cbd12ae-c043-47ca-bcb5-c6ac2ecf1436/router/0.log"
Sep 18 16:13:18.741620 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:18.741586    2561 scope.go:117] "RemoveContainer" containerID="b5261e67c064eb098434dea8b491d4024c1c9c7980fa469d84ad4328c1ea097d"
Sep 18 16:13:19.504923 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:19.504895    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-xn2fc_32803062-157c-4605-8474-a32715b72352/kube-storage-version-migrator-operator/1.log"
Sep 18 16:13:19.505783 ip-10-0-133-137 kubenswrapper[2561]: I0918 16:13:19.505762    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-xn2fc_32803062-157c-4605-8474-a32715b72352/kube-storage-version-migrator-operator/0.log"